Package com.digitalpebble.stormcrawler.filtering.basic

Class BasicURLFilter

java.lang.Object

com.digitalpebble.stormcrawler.util.AbstractConfigurable

com.digitalpebble.stormcrawler.filtering.URLFilter

com.digitalpebble.stormcrawler.filtering.basic.BasicURLFilter

All Implemented Interfaces:

Configurable

public class BasicURLFilter
extends URLFilter

Simple URL filters : can be used early in the filtering chain

Constructor Summary

Constructors

Constructor	Description
BasicURLFilter()

Method Summary

Modifier and Type	Method	Description
void	configure(@NotNull Map<String,Object> stormConf, @NotNull com.fasterxml.jackson.databind.JsonNode filterParams)	Called when this filter is being initialized
@Nullable String	filter(@Nullable URL sourceUrl, @Nullable Metadata sourceMetadata, @NotNull String urlToFilter)	Returns null if the URL is to be removed or a normalised representation which can correspond to the input URL
String	filterPathRepet(String urlToFilter)

Methods inherited from class com.digitalpebble.stormcrawler.util.AbstractConfigurable

configure, getName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BasicURLFilter

public BasicURLFilter()

Method Detail

filter

@Nullable
public @Nullable String filter(@Nullable
                               @Nullable URL sourceUrl,
                               @Nullable
                               @Nullable Metadata sourceMetadata,
                               @NotNull
                               @NotNull String urlToFilter)

Description copied from class: URLFilter

Returns null if the URL is to be removed or a normalised representation which can correspond to the input URL

Specified by:

filter in class URLFilter

Parameters:

sourceUrl - the URL of the page where the URL was found. Can be null.

sourceMetadata - the metadata collected for the page

urlToFilter - the URL to be filtered

Returns:

null if the url is to be removed or a normalised representation which can correspond to the input URL

filterPathRepet

public final String filterPathRepet(String urlToFilter)

configure

public void configure(@NotNull
                      @NotNull Map<String,Object> stormConf,
                      @NotNull
                      @NotNull com.fasterxml.jackson.databind.JsonNode filterParams)

Description copied from interface: Configurable

Called when this filter is being initialized

Parameters:

stormConf - The Storm configuration used for the configurable

filterParams - the filter specific configuration. Never null