org.archive.modules.writer

Class WARCWriterProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.modules.writer.WriterPoolProcessor

org.archive.modules.writer.BaseWARCWriterProcessor

org.archive.modules.writer.WARCWriterProcessor

All Implemented Interfaces:

Checkpointable, org.archive.io.warc.WARCWriterPoolSettings, WriterPoolSettings, HasKeyedProperties, org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

Deprecated.

in favor of WARCWriterChainProcessor

@Deprecated
public class WARCWriterProcessor
extends BaseWARCWriterProcessor
implements org.archive.io.warc.WARCWriterPoolSettings

WARCWriterProcessor. Intends to follow the WARC/1.0 specification.

TODO: Remove ANVLRecord. Rename NameValue or use RFC822 (commons-httpclient?) or find something else.

Author:

stack

Field Summary

Fields inherited from class org.archive.modules.writer.BaseWARCWriterProcessor

generator, stats, urlsWritten

Fields inherited from class org.archive.modules.writer.WriterPoolProcessor

ANNOTATION_UNWRITTEN, compress, directory, frequentFlushes, maxFileSizeBytes, maxTotalBytesToWrite, maxWaitForIdleMs, poolMaxActive, prefix, serverCache, skipIdenticalDigests, startNewFilesOnCheckpoint, storePaths, template, writeBufferSize

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
WARCWriterProcessor() Deprecated.

Method Summary

Modifier and Type	Method and Description
protected void	fromCheckpointJson(org.json.JSONObject json) Deprecated. Restore internal state from JSONObject stored at earlier checkpoint-time.
boolean	getWriteMetadata() Deprecated.
boolean	getWriteRequests() Deprecated.
protected ProcessResult	innerProcessResult(CrawlURI curi) Deprecated. Writes a CrawlURI and its associated data to store file.
protected URI	qualifyRecordID(URI base, String key, String value) Deprecated.
protected void	saveHeader(CrawlURI curi, org.archive.util.anvl.ANVLRecord warcHeaders, String origName, String newName) Deprecated. Saves a header from the given HTTP operation into the provider headers under a new name
void	setWriteMetadata(boolean writeMetadata) Deprecated. Whether to write 'metadata' type records.
void	setWriteRequests(boolean writeRequests) Deprecated. Whether to write 'request' type records.
void	setWriteRevisitForIdenticalDigests(boolean writeRevisits) Deprecated.
void	setWriteRevisitForNotModified(boolean writeRevisits) Deprecated.
protected org.json.JSONObject	toCheckpointJson() Deprecated. Return a JSONObject of current stat that can be consulted on recovery to restore necessary values.
protected ProcessResult	write(String lowerCaseScheme, CrawlURI curi) Deprecated.
protected void	writeDnsRecords(CrawlURI curi, org.archive.io.warc.WARCWriter w, URI baseid, String timestamp) Deprecated.
protected URI	writeFtpControlConversation(org.archive.io.warc.WARCWriter w, String timestamp, URI baseid, CrawlURI curi, org.archive.util.anvl.ANVLRecord headers, String controlConversation) Deprecated.
protected void	writeFtpRecords(org.archive.io.warc.WARCWriter w, CrawlURI curi, URI baseid, String timestamp) Deprecated.
protected void	writeHttpRecords(CrawlURI curi, org.archive.io.warc.WARCWriter w, URI baseid, String timestamp) Deprecated.
protected URI	writeMetadata(org.archive.io.warc.WARCWriter w, String timestamp, URI baseid, CrawlURI curi, org.archive.util.anvl.ANVLRecord namedFields) Deprecated.
protected URI	writeRequest(org.archive.io.warc.WARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, org.archive.util.anvl.ANVLRecord namedFields) Deprecated.
protected URI	writeResource(org.archive.io.warc.WARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, org.archive.util.anvl.ANVLRecord namedFields) Deprecated.
protected URI	writeResponse(org.archive.io.warc.WARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, org.archive.util.anvl.ANVLRecord suppliedFields) Deprecated.
protected URI	writeRevisit(org.archive.io.warc.WARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, org.archive.util.anvl.ANVLRecord headers) Deprecated.
protected URI	writeRevisit(org.archive.io.warc.WARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, org.archive.util.anvl.ANVLRecord headers, long contentLength) Deprecated.
protected void	writeWhoisRecords(org.archive.io.warc.WARCWriter w, CrawlURI curi, URI baseid, String timestamp) Deprecated.

Methods inherited from class org.archive.modules.writer.BaseWARCWriterProcessor

addIfNotBlank, addStats, copyStats, getDefaultMaxFileSize, getDefaultStorePaths, getMetadata, getRecordID, getRecordIDGenerator, getStats, report, setRecordIDGenerator, setupPool, updateMetadataAfterWrite

Methods inherited from class org.archive.modules.writer.WriterPoolProcessor

addTotalBytesWritten, calcOutputDirs, checkBytesWritten, copyForwardWriteTagIfDupe, doCheckpoint, getCompress, getDirectory, getFrequentFlushes, getHostAddress, getMaxFileSizeBytes, getMaxTotalBytesToWrite, getMaxWaitForIdleMs, getMetadataProvider, getPool, getPoolMaxActive, getPrefix, getSerialNo, getServerCache, getSkipIdenticalDigests, getStartNewFilesOnCheckpoint, getStorePaths, getTemplate, getTotalBytesWritten, getWriteBufferSize, innerProcess, innerRejectProcess, setCompress, setDirectory, setFrequentFlushes, setMaxFileSizeBytes, setMaxTotalBytesToWrite, setMaxWaitForIdleMs, setMetadataProvider, setPool, setPoolMaxActive, setPrefix, setServerCache, setSkipIdenticalDigests, setStartNewFilesOnCheckpoint, setStorePaths, setTemplate, setTotalBytesWritten, setWriteBufferSize, shouldProcess, shouldWrite, start, stop

Methods inherited from class org.archive.modules.Processor

finishCheckpoint, flattenVia, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, isRunning, isSuccess, process, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.archive.io.warc.WARCWriterPoolSettings

getRecordIDGenerator

Methods inherited from interface org.archive.io.WriterPoolSettings

calcOutputDirs, getCompress, getFrequentFlushes, getMaxFileSizeBytes, getMetadata, getPrefix, getTemplate, getWriteBufferSize

Methods inherited from interface org.springframework.context.Lifecycle

isRunning

Methods inherited from interface org.archive.checkpointing.Checkpointable

finishCheckpoint, setRecoveryCheckpoint, startCheckpoint

Constructor Detail

WARCWriterProcessor

public WARCWriterProcessor()

Deprecated.

Method Detail

getWriteRequests

public boolean getWriteRequests()

Deprecated.

setWriteRequests

public void setWriteRequests(boolean writeRequests)

Deprecated.

Whether to write 'request' type records. Default is true.

getWriteMetadata

public boolean getWriteMetadata()

Deprecated.

setWriteMetadata

public void setWriteMetadata(boolean writeMetadata)

Deprecated.

Whether to write 'metadata' type records. Default is true.

setWriteRevisitForIdenticalDigests

@Deprecated
public void setWriteRevisitForIdenticalDigests(boolean writeRevisits)

Deprecated.

setWriteRevisitForNotModified

@Deprecated
public void setWriteRevisitForNotModified(boolean writeRevisits)

Deprecated.

innerProcessResult

protected ProcessResult innerProcessResult(CrawlURI curi)

Deprecated.

Writes a CrawlURI and its associated data to store file. Currently this method understands the following uri types: dns, http, and https.

Specified by:

innerProcessResult in class WriterPoolProcessor

Parameters:

curi - CrawlURI to process.

write

protected ProcessResult write(String lowerCaseScheme,
                              CrawlURI curi)
                       throws IOException

Deprecated.

Throws:

IOException

writeDnsRecords

protected void writeDnsRecords(CrawlURI curi,
                               org.archive.io.warc.WARCWriter w,
                               URI baseid,
                               String timestamp)
                        throws IOException

Deprecated.

Throws:

IOException

writeWhoisRecords

protected void writeWhoisRecords(org.archive.io.warc.WARCWriter w,
                                 CrawlURI curi,
                                 URI baseid,
                                 String timestamp)
                          throws IOException

Deprecated.

Throws:

IOException

writeHttpRecords

protected void writeHttpRecords(CrawlURI curi,
                                org.archive.io.warc.WARCWriter w,
                                URI baseid,
                                String timestamp)
                         throws IOException

Deprecated.

Throws:

IOException

writeFtpRecords

protected void writeFtpRecords(org.archive.io.warc.WARCWriter w,
                               CrawlURI curi,
                               URI baseid,
                               String timestamp)
                        throws IOException

Deprecated.

Throws:

IOException

writeFtpControlConversation

protected URI writeFtpControlConversation(org.archive.io.warc.WARCWriter w,
                                          String timestamp,
                                          URI baseid,
                                          CrawlURI curi,
                                          org.archive.util.anvl.ANVLRecord headers,
                                          String controlConversation)
                                   throws IOException

Deprecated.

Throws:

IOException

writeRequest

protected URI writeRequest(org.archive.io.warc.WARCWriter w,
                           String timestamp,
                           String mimetype,
                           URI baseid,
                           CrawlURI curi,
                           org.archive.util.anvl.ANVLRecord namedFields)
                    throws IOException

Deprecated.

Throws:

IOException

writeResponse

protected URI writeResponse(org.archive.io.warc.WARCWriter w,
                            String timestamp,
                            String mimetype,
                            URI baseid,
                            CrawlURI curi,
                            org.archive.util.anvl.ANVLRecord suppliedFields)
                     throws IOException

Deprecated.

Throws:

IOException

writeResource

protected URI writeResource(org.archive.io.warc.WARCWriter w,
                            String timestamp,
                            String mimetype,
                            URI baseid,
                            CrawlURI curi,
                            org.archive.util.anvl.ANVLRecord namedFields)
                     throws IOException

Deprecated.

Throws:

IOException

writeRevisit

protected URI writeRevisit(org.archive.io.warc.WARCWriter w,
                           String timestamp,
                           String mimetype,
                           URI baseid,
                           CrawlURI curi,
                           org.archive.util.anvl.ANVLRecord headers)
                    throws IOException

Deprecated.

Throws:

IOException

writeRevisit

protected URI writeRevisit(org.archive.io.warc.WARCWriter w,
                           String timestamp,
                           String mimetype,
                           URI baseid,
                           CrawlURI curi,
                           org.archive.util.anvl.ANVLRecord headers,
                           long contentLength)
                    throws IOException

Deprecated.

Throws:

IOException

saveHeader

protected void saveHeader(CrawlURI curi,
                          org.archive.util.anvl.ANVLRecord warcHeaders,
                          String origName,
                          String newName)

Deprecated.

Saves a header from the given HTTP operation into the provider headers under a new name

writeMetadata

protected URI writeMetadata(org.archive.io.warc.WARCWriter w,
                            String timestamp,
                            URI baseid,
                            CrawlURI curi,
                            org.archive.util.anvl.ANVLRecord namedFields)
                     throws IOException

Deprecated.

Throws:

IOException

qualifyRecordID

protected URI qualifyRecordID(URI base,
                              String key,
                              String value)
                       throws IOException

Deprecated.

Throws:

IOException

toCheckpointJson

protected org.json.JSONObject toCheckpointJson()
                                        throws org.json.JSONException

Deprecated.

Description copied from class: Processor

Return a JSONObject of current stat that can be consulted on recovery to restore necessary values.

Overrides:

toCheckpointJson in class WriterPoolProcessor

Returns:

JSONObject

Throws:

org.json.JSONException

fromCheckpointJson

protected void fromCheckpointJson(org.json.JSONObject json)
                           throws org.json.JSONException

Deprecated.

Description copied from class: Processor

Restore internal state from JSONObject stored at earlier checkpoint-time.

Overrides:

fromCheckpointJson in class WriterPoolProcessor

Parameters:

json - JSONObject

Throws:

org.json.JSONException