Package com.digitalpebble.stormcrawler.protocol

Class RobotRules

  • All Implemented Interfaces:
    Serializable
    public class RobotRules
extends crawlercommons.robots.BaseRobotRules
    Wrapper for BaseRobotRules which tracks the number of requests and length of the responses needed to get the rules. If the array returned by getContentLengthFetched() is empty, then the rules were obtained from the cache.
    See Also:
    Serialized Form

    • Constructor Detail

      • RobotRules

        public RobotRules​(crawlercommons.robots.BaseRobotRules base)

    • Method Detail

      • isAllowed

        public boolean isAllowed​(String url)
        Specified by:
        isAllowed in class crawlercommons.robots.BaseRobotRules

      • isAllowAll

        public boolean isAllowAll()
        Specified by:
        isAllowAll in class crawlercommons.robots.BaseRobotRules

      • isAllowNone

        public boolean isAllowNone()
        Specified by:
        isAllowNone in class crawlercommons.robots.BaseRobotRules

      • getContentLengthFetched

        public int[] getContentLengthFetched()
        Returns the number of bytes fetched per request when not cached *

      • setContentLengthFetched

        public void setContentLengthFetched​(int[] bytesFetched)
        Returns the number of bytes fetched per request when not cached *

      • getCrawlDelay

        public long getCrawlDelay()
        Overrides:
        getCrawlDelay in class crawlercommons.robots.BaseRobotRules

      • isDeferVisits

        public boolean isDeferVisits()
        Overrides:
        isDeferVisits in class crawlercommons.robots.BaseRobotRules

      • getSitemaps

        public List<String> getSitemaps()
        Overrides:
        getSitemaps in class crawlercommons.robots.BaseRobotRules

      • hashCode

        public int hashCode()
        Overrides:
        hashCode in class crawlercommons.robots.BaseRobotRules

      • equals

        public boolean equals​(Object obj)
        Overrides:
        equals in class crawlercommons.robots.BaseRobotRules

      • setCrawlDelay

        public void setCrawlDelay​(long crawlDelay)
        Overrides:
        setCrawlDelay in class crawlercommons.robots.BaseRobotRules

      • setDeferVisits

        public void setDeferVisits​(boolean deferVisits)
        Overrides:
        setDeferVisits in class crawlercommons.robots.BaseRobotRules

      • addSitemap

        public void addSitemap​(String sitemap)
        Overrides:
        addSitemap in class crawlercommons.robots.BaseRobotRules