org.archive.modules.net

Class ObeyRobotsPolicy

java.lang.Object

org.archive.modules.net.RobotsPolicy

org.archive.modules.net.ObeyRobotsPolicy

public class ObeyRobotsPolicy
extends RobotsPolicy

Classic obey-robots-as-declared policy.

Author:

gojomo

Field Summary

Fields

Modifier and Type	Field and Description
static RobotsPolicy	INSTANCE

Fields inherited from class org.archive.modules.net.RobotsPolicy

STANDARD_POLICIES

Constructor Summary

Constructors

Constructor and Description
ObeyRobotsPolicy()

Method Summary

Modifier and Type	Method and Description
boolean	allows(String userAgent, CrawlURI curi, Robotstxt robotstxt)
boolean	obeyMetaRobotsNofollow()

Methods inherited from class org.archive.modules.net.RobotsPolicy

getPathQuery

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INSTANCE

public static RobotsPolicy INSTANCE

Constructor Detail

ObeyRobotsPolicy

public ObeyRobotsPolicy()

Method Detail

allows

public boolean allows(String userAgent,
                      CrawlURI curi,
                      Robotstxt robotstxt)

Specified by:

allows in class RobotsPolicy

obeyMetaRobotsNofollow

public boolean obeyMetaRobotsNofollow()

Specified by:

obeyMetaRobotsNofollow in class RobotsPolicy