public class FetchHistoryProcessor extends Processor
Modifier and Type | Field and Description |
---|---|
protected int |
historyLength
Desired history array length.
|
Constructor and Description |
---|
FetchHistoryProcessor() |
Modifier and Type | Method and Description |
---|---|
int |
getHistoryLength() |
static boolean |
hasIdenticalDigest(CrawlURI curi)
Utility method for testing if a CrawlURI's last two history
entries (one being the most recent fetch) have identical
content-digest information.
|
protected HashMap<String,Object>[] |
historyRealloc(CrawlURI curi)
Get or create proper-sized history array
|
protected void |
innerProcess(CrawlURI puri)
Actually performs the process.
|
protected void |
saveHeader(CrawlURI curi,
Map<String,Object> map,
String key)
Save a header from the given HTTP operation into the Map.
|
void |
setHistoryLength(int length) |
protected boolean |
shouldProcess(CrawlURI curi)
Determines whether the given uri should be processed by this
processor.
|
doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson
public int getHistoryLength()
public void setHistoryLength(int length)
protected void innerProcess(CrawlURI puri) throws InterruptedException
Processor
Processor.getEnabled()
, the
Processor.getShouldProcessRule()
and the Processor.shouldProcess(CrawlURI)
tests.innerProcess
in class Processor
puri
- the URI to processInterruptedException
- if the thread is interruptedpublic static boolean hasIdenticalDigest(CrawlURI curi)
curi
- CrawlURI to testprotected HashMap<String,Object>[] historyRealloc(CrawlURI curi)
protected void saveHeader(CrawlURI curi, Map<String,Object> map, String key)
protected boolean shouldProcess(CrawlURI curi)
Processor
shouldProcess
in class Processor
curi
- the URI to testCopyright © 2003–2021 Internet Archive. All rights reserved.