Package org.archive.modules.net

Class ObeyRobotsPolicy

java.lang.Object

org.archive.modules.net.RobotsPolicy

org.archive.modules.net.ObeyRobotsPolicy

Direct Known Subclasses:

RobotsTxtOnlyPolicy

public class ObeyRobotsPolicy
extends RobotsPolicy

Classic obey-robots-as-declared policy.

Author:

gojomo

Field Summary

Fields

Modifier and Type	Field	Description
static RobotsPolicy	INSTANCE

Fields inherited from class org.archive.modules.net.RobotsPolicy

STANDARD_POLICIES

Constructor Summary

Constructors

Constructor	Description
ObeyRobotsPolicy()

Method Summary

Modifier and Type	Method	Description
boolean	allows(String userAgent, CrawlURI curi, Robotstxt robotstxt)
boolean	obeyMetaRobotsNofollow()

Methods inherited from class org.archive.modules.net.RobotsPolicy

getPathQuery

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

INSTANCE

public static RobotsPolicy INSTANCE

Constructor Details

ObeyRobotsPolicy

public ObeyRobotsPolicy()

Method Details

allows

public boolean allows(String userAgent,
 CrawlURI curi,
 Robotstxt robotstxt)

Specified by:

allows in class RobotsPolicy

obeyMetaRobotsNofollow

public boolean obeyMetaRobotsNofollow()

Specified by:

obeyMetaRobotsNofollow in class RobotsPolicy