Package com.digitalpebble.stormcrawler.filtering.robots
-
Class Summary Class Description RobotsFilter URLFilter which discards URLs based on the robots.txt directives.
Class | Description |
---|---|
RobotsFilter |
URLFilter which discards URLs based on the robots.txt directives.
|