Package com.digitalpebble.stormcrawler.parse.filter

Class XPathFilter

java.lang.Object

com.digitalpebble.stormcrawler.util.AbstractConfigurable

com.digitalpebble.stormcrawler.parse.ParseFilter

com.digitalpebble.stormcrawler.parse.filter.XPathFilter

All Implemented Interfaces:

Configurable

Direct Known Subclasses:

LinkParseFilter

public class XPathFilter
extends ParseFilter

Simple ParseFilter to illustrate and test the interface. Reads a XPATH pattern from the config file and stores the value as metadata

Field Summary

Fields

Modifier and Type	Field	Description
protected Map<String,List<com.digitalpebble.stormcrawler.parse.filter.XPathFilter.LabelledExpression>>	expressions

Constructor Summary

Constructors

Constructor	Description
XPathFilter()

Method Summary

Modifier and Type	Method	Description
void	configure(@NotNull Map<String,Object> stormConf, @NotNull com.fasterxml.jackson.databind.JsonNode filterParams)	Called when this filter is being initialized
void	filter(String URL, byte[] content, DocumentFragment doc, ParseResult parse)	Called when parsing a specific page
boolean	needsDOM()	Specifies whether this filter requires a DOM representation of the document

Methods inherited from class com.digitalpebble.stormcrawler.util.AbstractConfigurable

configure, getName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

expressions

protected final Map<String,List<com.digitalpebble.stormcrawler.parse.filter.XPathFilter.LabelledExpression>> expressions

Constructor Detail

XPathFilter

public XPathFilter()

Method Detail

filter

public void filter(String URL,
                   byte[] content,
                   DocumentFragment doc,
                   ParseResult parse)

Description copied from class: ParseFilter

Called when parsing a specific page

Specified by:

filter in class ParseFilter

Parameters:

URL - the URL of the page being parsed

content - the content being parsed

doc - the DOM tree resulting of the parsing of the content or null if ParseFilter.needsDOM() returns false

parse - the metadata to be updated with the resulting of the parsing

configure

public void configure(@NotNull
                      @NotNull Map<String,Object> stormConf,
                      @NotNull
                      @NotNull com.fasterxml.jackson.databind.JsonNode filterParams)

Description copied from interface: Configurable

Called when this filter is being initialized

Parameters:

stormConf - The Storm configuration used for the configurable

filterParams - the filter specific configuration. Never null

needsDOM

public boolean needsDOM()

Description copied from class: ParseFilter

Specifies whether this filter requires a DOM representation of the document

Overrides:

needsDOM in class ParseFilter

Returns:

trueif this needs a DOM representation of the document, false otherwise.