Package org.codelibs.fess.crawler.helper

Class RobotsTxtHelper

java.lang.Object

org.codelibs.fess.crawler.helper.RobotsTxtHelper

public class RobotsTxtHelper
extends Object

Robots.txt Specifications:

• https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

Author:

bowez, shinsuke

Field Summary

Fields

Modifier and Type	Field	Description
protected static final Pattern	ALLOW_RECORD
protected static final Pattern	CRAWL_DELAY_RECORD
protected static final Pattern	DISALLOW_RECORD
protected boolean	enabled
protected static final Pattern	SITEMAP_RECORD
protected static final Pattern	USER_AGENT_RECORD

Constructor Summary

Constructors

Constructor	Description
RobotsTxtHelper()

Method Summary

Modifier and Type	Method	Description
protected String	getValue(Pattern pattern, String line)
boolean	isEnabled()
RobotsTxt	parse(InputStream stream)
RobotsTxt	parse(InputStream stream, String charsetName)
void	setEnabled(boolean enabled)
protected String	stripComment(String line)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

USER_AGENT_RECORD

protected static final Pattern USER_AGENT_RECORD

DISALLOW_RECORD

protected static final Pattern DISALLOW_RECORD

ALLOW_RECORD

protected static final Pattern ALLOW_RECORD

CRAWL_DELAY_RECORD

protected static final Pattern CRAWL_DELAY_RECORD

SITEMAP_RECORD

protected static final Pattern SITEMAP_RECORD

enabled

protected boolean enabled

Constructor Details

RobotsTxtHelper

public RobotsTxtHelper()

Method Details

parse

public RobotsTxt parse(InputStream stream)

parse

public RobotsTxt parse(InputStream stream,
 String charsetName)

getValue

protected String getValue(Pattern pattern,
 String line)

stripComment

protected String stripComment(String line)

isEnabled

public boolean isEnabled()

setEnabled

public void setEnabled(boolean enabled)