Package com.digitalpebble.stormcrawler.elasticsearch.persistence

Class HybridSpout

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichSpout

com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout

com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout

com.digitalpebble.stormcrawler.elasticsearch.persistence.HybridSpout

All Implemented Interfaces:

EmptyQueueListener, Serializable, org.apache.storm.spout.ISpout, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichSpout, org.elasticsearch.action.ActionListener<org.elasticsearch.action.search.SearchResponse>

public class HybridSpout
extends AggregationSpout
implements EmptyQueueListener

Uses collapsing spouts to get an initial set of URLs and keys to query for and gets emptyQueue notifications from the URLBuffer to query ES for a specific key.

Since:

1.15

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

AbstractQueryingSpout.InProcessMap<K extends Object,V extends Object>

Nested classes/interfaces inherited from interface org.elasticsearch.action.ActionListener

org.elasticsearch.action.ActionListener.Delegating<Response extends Object,DelegateResponse extends Object>, org.elasticsearch.action.ActionListener.DelegatingActionListener<T extends Object>, org.elasticsearch.action.ActionListener.DelegatingFailureActionListener<T extends Object,R extends Object>, org.elasticsearch.action.ActionListener.MappedActionListener<Response extends Object,MappedResponse extends Object>, org.elasticsearch.action.ActionListener.RunAfterActionListener<T extends Object>, org.elasticsearch.action.ActionListener.RunBeforeActionListener<T extends Object>

Field Summary

Fields

Modifier and Type	Field	Description
protected static String	RELOADPARAMNAME

Fields inherited from class com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout

currentBuckets

Fields inherited from class com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout

bucketSortField, client, ESBoltType, ESStatusBucketFieldParamName, ESStatusBucketSortFieldParamName, ESStatusFilterParamName, ESStatusGlobalSortFieldParamName, ESStatusIndexNameParamName, ESStatusMaxBucketParamName, ESStatusMaxURLsParamName, ESStatusQueryTimeoutParamName, filterQueries, indexName, logIdprefix, maxBucketNum, maxURLsPerBucket, partitionField, queryDate, queryTimeout, shardID, totalSortField

Fields inherited from class com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

_collector, beingProcessed, buffer, eventCounter, isInQuery, lastTimeResetToNOW, maxDelayBetweenQueries, minDelayBetweenQueries, queryTimes, resetFetchDateAfterNSecs, resetFetchDateParamName, StatusMaxDelayParamName, StatusMinDelayParamName, StatusTTLPurgatory

Constructor Summary

Constructors

Constructor	Description
HybridSpout()

Method Summary

Modifier and Type	Method	Description
void	emptyQueue(String queueName)
void	onResponse(org.elasticsearch.action.search.SearchResponse response)
void	open(Map<String,Object> stormConf, org.apache.storm.task.TopologyContext context, org.apache.storm.spout.SpoutOutputCollector collector)
protected void	sortValuesForKey(String key, Object[] sortValues)

Methods inherited from class com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout

onFailure, populateBuffer

Methods inherited from class com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout

ack, addHitToBuffer, close, fail, fromKeyValues

Methods inherited from class com.digitalpebble.stormcrawler.persistence.AbstractQueryingSpout

activate, deactivate, declareOutputFields, getTimeLastQuerySent, markQueryReceivedNow, nextTuple

Methods inherited from class org.apache.storm.topology.base.BaseComponent

getComponentConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.elasticsearch.action.ActionListener

delegateFailure, delegateResponse, map

Methods inherited from interface org.apache.storm.topology.IComponent

getComponentConfiguration

Field Detail

RELOADPARAMNAME

protected static final String RELOADPARAMNAME

See Also:

Constant Field Values

Constructor Detail

HybridSpout

public HybridSpout()

Method Detail

open

public void open(Map<String,Object> stormConf,
                 org.apache.storm.task.TopologyContext context,
                 org.apache.storm.spout.SpoutOutputCollector collector)

Specified by:

open in interface org.apache.storm.spout.ISpout

Overrides:

open in class AggregationSpout

emptyQueue

public void emptyQueue(String queueName)

Specified by:

emptyQueue in interface EmptyQueueListener

onResponse

public void onResponse(org.elasticsearch.action.search.SearchResponse response)

Specified by:

onResponse in interface org.elasticsearch.action.ActionListener<org.elasticsearch.action.search.SearchResponse>

Overrides:

onResponse in class AggregationSpout

sortValuesForKey

protected void sortValuesForKey(String key,
                                Object[] sortValues)

Overrides:

sortValuesForKey in class AggregationSpout