Package com.digitalpebble.stormcrawler.protocol
-
Interface Summary Interface Description Protocol -
Class Summary Class Description AbstractHttpProtocol AbstractHttpProtocol.KeyValue DelegatorProtocol Protocol implementation that enables selection from a collection of sub-protocols using filters based on each call's metadata and URL.HttpHeaders A collection of HTTP header names and utilities around header values.HttpRobotRulesParser This class is used for parsing robots for urls belonging to HTTP protocol.ProtocolFactory ProtocolResponse RobotRules Wrapper for BaseRobotRules which tracks the number of requests and length of the responses needed to get the rules.RobotRulesParser This class uses crawler-commons for handling the parsing ofrobots.txt
files. -
Enum Summary Enum Description ProtocolResponse.TrimmedContentReason Enum of reasons which may cause that protocol content is trimmed.