Package org.archive.crawler.postprocessor

package org.archive.crawler.postprocessor
  • Classes
    Class
    Description
    CandidatesProcessor
    Processor which sends all candidate outlinks through the CandidateChain, scheduling those with non-negative status codes to the frontier.
    DispositionProcessor
    A step, late in the processing of a CrawlURI, for marking-up the CrawlURI with values to affect frontier disposition, and updating information that may have been affected by the fetch.
    LowDiskPauseProcessor
    Deprecated.
    Is highly system dependent.
    ReschedulingProcessor
    The most simple forced-rescheduling step possible: use a local setting (perhaps overlaid to vary based on the URI) to set an exact future reschedule time, as a delay from now.
    SupplementaryLinksScoper
    Run CrawlURI links carried in the passed CrawlURI through a filter and 'handle' rejections.