Package org.archive.crawler.postprocessor

Class DispositionProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.crawler.postprocessor.DispositionProcessor

All Implemented Interfaces:

org.archive.checkpointing.Checkpointable, org.archive.spring.HasKeyedProperties, org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class DispositionProcessor
extends Processor

A step, late in the processing of a CrawlURI, for marking-up the CrawlURI with values to affect frontier disposition, and updating information that may have been affected by the fetch. This includes robots info and other stats. (Formerly called CrawlStateUpdater, when it did less.)

Version:

$Date$, $Revision$

Author:

gojomo

Field Summary

Fields

Modifier and Type	Field	Description
protected CrawlMetadata	metadata
protected ServerCache	serverCache

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor	Description
DispositionProcessor()

Method Summary

Modifier and Type	Method	Description
float	getDelayFactor()
boolean	getForceRetire()
int	getMaxDelayMs()
int	getMaxPerHostBandwidthUsageKbSec()
CrawlMetadata	getMetadata()
int	getMinDelayMs()
int	getRespectCrawlDelayUpToSeconds()
ServerCache	getServerCache()
protected void	innerProcess(CrawlURI curi)
protected long	politenessDelayFor(CrawlURI curi)	Update any scheduling structures with the new information in this CrawlURI.
void	setDelayFactor(float factor)	How many multiples of last fetch elapsed time to wait before recontacting same server.
void	setForceRetire(boolean force)	Whether to set a CrawlURI's force-retired directive, retiring its queue when it finishes.
void	setMaxDelayMs(int maxDelay)	never wait more than this long, regardless of multiple
void	setMaxPerHostBandwidthUsageKbSec(int max)	maximum per-host bandwidth usage
void	setMetadata(CrawlMetadata provider)	Auto-discovered module providing configured (or overridden) User-Agent value and RobotsHonoringPolicy
void	setMinDelayMs(int minDelay)	always wait this long after one completion before recontacting same server, regardless of multiple
void	setRespectCrawlDelayUpToSeconds(int respect)	Whether to respect a 'Crawl-Delay' (in seconds) given in a site's robots.txt
void	setServerCache(ServerCache serverCache)
protected boolean	shouldProcess(CrawlURI puri)

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

serverCache

protected ServerCache serverCache

metadata

protected CrawlMetadata metadata

Constructor Details

DispositionProcessor

public DispositionProcessor()

Method Details

getServerCache

public ServerCache getServerCache()

setServerCache

@Autowired
public void setServerCache(ServerCache serverCache)

getDelayFactor

public float getDelayFactor()

setDelayFactor

public void setDelayFactor(float factor)

How many multiples of last fetch elapsed time to wait before recontacting same server.

getMinDelayMs

public int getMinDelayMs()

setMinDelayMs

public void setMinDelayMs(int minDelay)

always wait this long after one completion before recontacting same server, regardless of multiple

getRespectCrawlDelayUpToSeconds

public int getRespectCrawlDelayUpToSeconds()

setRespectCrawlDelayUpToSeconds

public void setRespectCrawlDelayUpToSeconds(int respect)

Whether to respect a 'Crawl-Delay' (in seconds) given in a site's robots.txt

getMaxDelayMs

public int getMaxDelayMs()

setMaxDelayMs

public void setMaxDelayMs(int maxDelay)

never wait more than this long, regardless of multiple

getMaxPerHostBandwidthUsageKbSec

public int getMaxPerHostBandwidthUsageKbSec()

setMaxPerHostBandwidthUsageKbSec

public void setMaxPerHostBandwidthUsageKbSec(int max)

maximum per-host bandwidth usage

getForceRetire

public boolean getForceRetire()

setForceRetire

public void setForceRetire(boolean force)

Whether to set a CrawlURI's force-retired directive, retiring its queue when it finishes. Mainly intended for URI-specific overlay settings; setting true globally will just retire all queues after they offer one URI, rapidly ending a crawl.

getMetadata

public CrawlMetadata getMetadata()

setMetadata

@Autowired
public void setMetadata(CrawlMetadata provider)

Auto-discovered module providing configured (or overridden) User-Agent value and RobotsHonoringPolicy

shouldProcess

protected boolean shouldProcess(CrawlURI puri)

Specified by:

shouldProcess in class Processor

innerProcess

protected void innerProcess(CrawlURI curi)

Specified by:

innerProcess in class Processor

politenessDelayFor

protected long politenessDelayFor(CrawlURI curi)

Update any scheduling structures with the new information in this CrawlURI. Chiefly means make necessary arrangements for no other URIs at the same host to be visited within the appropriate politeness window.

Parameters:

curi - The CrawlURI

Returns:

millisecond politeness delay