Package com.digitalpebble.stormcrawler.elasticsearch.parse.filter

Class JSONResourceWrapper

java.lang.Object

com.digitalpebble.stormcrawler.util.AbstractConfigurable

com.digitalpebble.stormcrawler.parse.ParseFilter

com.digitalpebble.stormcrawler.elasticsearch.parse.filter.JSONResourceWrapper

All Implemented Interfaces:

Configurable

public class JSONResourceWrapper
extends ParseFilter

Wraps a ParseFilter whose resources are in a JSON file that can be stored in ES. The benefit of doing this is that the resources can be refreshed automatically and modified without having to recompile the jar and restart the topology. The connection to ES is done via the config and uses a new bolt type 'config'.

The configuration of the delegate is done in the parsefilters.json as usual.

  {
     "class": "com.digitalpebble.stormcrawler.elasticsearch.parse.filter.JSONResourceWrapper",
     "name": "ESCollectionTagger",
     "params": {
         "refresh": "60",
         "delegate": {
             "class": "com.digitalpebble.stormcrawler.parse.filter.CollectionTagger",
             "params": {
                 "file": "collections.json"
             }
         }
     }
  }
 

The resource file can be pushed to ES with

  curl -XPUT "$ESHOST/config/_create/collections.json" -H 'Content-Type: application/json' -d @src/main/resources/collections.json
 

Constructor Summary

Constructors

Constructor	Description
JSONResourceWrapper()

Method Summary

Modifier and Type	Method	Description
void	configure(@NotNull Map<String,Object> stormConf, @NotNull com.fasterxml.jackson.databind.JsonNode filterParams)
void	filter(String URL, byte[] content, DocumentFragment doc, ParseResult parse)

Methods inherited from class com.digitalpebble.stormcrawler.parse.ParseFilter

needsDOM

Methods inherited from class com.digitalpebble.stormcrawler.util.AbstractConfigurable

configure, getName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JSONResourceWrapper

public JSONResourceWrapper()

Method Detail

configure

public void configure(@NotNull
                      @NotNull Map<String,Object> stormConf,
                      @NotNull
                      @NotNull com.fasterxml.jackson.databind.JsonNode filterParams)

filter

public void filter(String URL,
                   byte[] content,
                   DocumentFragment doc,
                   ParseResult parse)

Specified by:

filter in class ParseFilter