Package com.digitalpebble.stormcrawler.util

Interface Configurable

All Known Subinterfaces:

JSoupFilter

All Known Implementing Classes:

AbstractConfigurable, BasicURLFilter, BasicURLNormalizer, CollectionTagger, CommaSeparatedToMultivaluedMetadata, DebugParseFilter, DomainParseFilter, FastURLFilter, HostURLFilter, JSoupFilters, LDJsonParseFilter, LDJsonParseFilter, LinkParseFilter, LinkParseFilter, MaxDepthFilter, MD5SignatureParseFilter, MetadataFilter, MimeTypeNormalization, NavigationFilter, NavigationFilters, ParseFilter, ParseFilters, RegexURLFilter, RegexURLFilterBase, RegexURLNormalizer, RobotsFilter, SelfURLFilter, SitemapFilter, URLFilter, URLFilters, XPathFilter, XPathFilter

public interface Configurable

An interface marking the implementing class as initializeable and configurable via createConfiguredInstance(String, Class, Map, JsonNode) The implementing class HAS to implement an empty constructor.

Method Summary

Modifier and Type	Method	Description
default void	configure(@NotNull Map<String,Object> stormConf, @NotNull com.fasterxml.jackson.databind.JsonNode filterParams)	Called when this filter is being initialized
static <T extends Configurable> @NotNull List<@NotNull T>	configure(@NotNull Map<String,Object> stormConf, @NotNull com.fasterxml.jackson.databind.JsonNode filtersConf, @NotNull Class<T> filterClass, @NotNull String callingClass)	Deprecated. Replace with createConfiguredInstance(Class, Class, Map, JsonNode) or createConfiguredInstance(String, Class, Map, JsonNode)
default void	configure(@NotNull Map<String,Object> stormConf, @NotNull com.fasterxml.jackson.databind.JsonNode filterParams, @NotNull String filterName)	Called when this filter is being initialized
static <T extends Configurable> @NotNull List<@NotNull T>	createConfiguredInstance(@NotNull Class<?> caller, @NotNull Class<T> filterClass, @NotNull Map<String,Object> stormConf, @NotNull com.fasterxml.jackson.databind.JsonNode filtersConf)	Calls createConfiguredInstance(String, Class, Map, JsonNode) with caller.getName() for configName.
static <T extends Configurable> @NotNull List<@NotNull T>	createConfiguredInstance(@NotNull String configName, @NotNull Class<T> filterClass, @NotNull Map<String,Object> stormConf, @NotNull com.fasterxml.jackson.databind.JsonNode filtersConf)	Used by classes like URLFilters and ParseFilters to load the configuration of utilized filters from the provided JSON config.
String	getName()

Method Detail

getName

String getName()

configure

default void configure(@NotNull
                       @NotNull Map<String,Object> stormConf,
                       @NotNull
                       @NotNull com.fasterxml.jackson.databind.JsonNode filterParams)

Called when this filter is being initialized

Parameters:

stormConf - The Storm configuration used for the configurable

filterParams - the filter specific configuration. Never null

configure

default void configure(@NotNull
                       @NotNull Map<String,Object> stormConf,
                       @NotNull
                       @NotNull com.fasterxml.jackson.databind.JsonNode filterParams,
                       @NotNull
                       @NotNull String filterName)

Called when this filter is being initialized

Parameters:

stormConf - The Storm configuration used for the configurable

filterParams - the filter specific configuration. Never null

filterName - The filter name.

createConfiguredInstance

@NotNull
static <T extends Configurable> @NotNull List<@NotNull T> createConfiguredInstance(@NotNull
                                                                                   @NotNull Class<?> caller,
                                                                                   @NotNull
                                                                                   @NotNull Class<T> filterClass,
                                                                                   @NotNull
                                                                                   @NotNull Map<String,Object> stormConf,
                                                                                   @NotNull
                                                                                   @NotNull com.fasterxml.jackson.databind.JsonNode filtersConf)

Calls createConfiguredInstance(String, Class, Map, JsonNode) with caller.getName() for configName.

See Also:

for more information.

createConfiguredInstance

@NotNull
static <T extends Configurable> @NotNull List<@NotNull T> createConfiguredInstance(@NotNull
                                                                                   @NotNull String configName,
                                                                                   @NotNull
                                                                                   @NotNull Class<T> filterClass,
                                                                                   @NotNull
                                                                                   @NotNull Map<String,Object> stormConf,
                                                                                   @NotNull
                                                                                   @NotNull com.fasterxml.jackson.databind.JsonNode filtersConf)

Used by classes like URLFilters and ParseFilters to load the configuration of utilized filters from the provided JSON config.

The functions searches for a childNode in filtersConf with the given configName. If the childNode is found it initializes all elements in the list provided by the filtersConf and initialized them as filterClass.

The following snippet shows the JSON-Schema for a config file, if the config file does not meet the schema, the function fails.

 {
   "$id": "https://stormcrawler.net/schemas/configurable/config",
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "type": "object",
   "properties": {
     <configName>: {
       "type": "array",
       "contains": {
         "type": "object",
         "properties": {
           "name": {
             "type": "string",
             "default": "<unnamed>"
           },
           "class": {
             "type": "string"
           },
           "properties": {
             "type": "array",
             "default": null
           }
         },
         "required": [
           "class"
         ]
       }
     }
   }
 }
 

configure

@Deprecated
@NotNull
static <T extends Configurable> @NotNull List<@NotNull T> configure(@NotNull
                                                                    @NotNull Map<String,Object> stormConf,
                                                                    @NotNull
                                                                    @NotNull com.fasterxml.jackson.databind.JsonNode filtersConf,
                                                                    @NotNull
                                                                    @NotNull Class<T> filterClass,
                                                                    @NotNull
                                                                    @NotNull String callingClass)

Deprecated.

Replace with createConfiguredInstance(Class, Class, Map, JsonNode) or createConfiguredInstance(String, Class, Map, JsonNode)