Package org.archive.modules.recrawl

Class ContentDigestHistoryLoader

java.lang.Object

org.archive.modules.Processor

org.archive.modules.recrawl.ContentDigestHistoryLoader

All Implemented Interfaces:

org.archive.checkpointing.Checkpointable, org.archive.spring.HasKeyedProperties, org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class ContentDigestHistoryLoader
extends Processor

Field Summary

Fields

Modifier and Type	Field	Description
protected AbstractContentDigestHistory	contentDigestHistory

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor	Description
ContentDigestHistoryLoader()

Method Summary

Modifier and Type	Method	Description
protected void	innerProcess(CrawlURI curi)	Actually performs the process.
void	setContentDigestHistory(AbstractContentDigestHistory contentDigestHistory)
protected boolean	shouldProcess(CrawlURI uri)	Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

contentDigestHistory

protected AbstractContentDigestHistory contentDigestHistory

Constructor Details

ContentDigestHistoryLoader

public ContentDigestHistoryLoader()

Method Details

setContentDigestHistory

@Autowired
public void setContentDigestHistory(AbstractContentDigestHistory contentDigestHistory)

shouldProcess

protected boolean shouldProcess(CrawlURI uri)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

uri - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI curi)
                     throws InterruptedException

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the Processor.getEnabled(), the Processor.getShouldProcessRule() and the Processor.shouldProcess(CrawlURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

curi - the URI to process

Throws:

InterruptedException - if the thread is interrupted