Package com.digitalpebble.stormcrawler.bolt

Class FetcherBolt

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichBolt

com.digitalpebble.stormcrawler.bolt.StatusEmitterBolt

com.digitalpebble.stormcrawler.bolt.FetcherBolt

All Implemented Interfaces:

Serializable, org.apache.storm.task.IBolt, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichBolt

public class FetcherBolt
extends StatusEmitterBolt

A multithreaded, queue-based fetcher adapted from Apache Nutch. Enforces the politeness and handles the fetching threads itself.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
static String	QUEUED_TIMEOUT_PARAM_KEY	Acks URLs which have spent too much time in the queue, should be set to a value equals to the topology timeout

Fields inherited from class com.digitalpebble.stormcrawler.bolt.StatusEmitterBolt

collector

Constructor Summary

Constructors

Constructor	Description
FetcherBolt()

Method Summary

Modifier and Type	Method	Description
void	cleanup()
void	declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)
void	execute(org.apache.storm.tuple.Tuple input)
Map<String,Object>	getComponentConfiguration()
void	prepare(Map<String,Object> stormConf, org.apache.storm.task.TopologyContext context, org.apache.storm.task.OutputCollector collector)

Methods inherited from class com.digitalpebble.stormcrawler.bolt.StatusEmitterBolt

allowRedirs, emitOutlink, filterOutlink

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

QUEUED_TIMEOUT_PARAM_KEY

public static final String QUEUED_TIMEOUT_PARAM_KEY

Acks URLs which have spent too much time in the queue, should be set to a value equals to the topology timeout

See Also:

Constant Field Values

Constructor Detail

FetcherBolt

public FetcherBolt()

Method Detail

getComponentConfiguration

public Map<String,Object> getComponentConfiguration()

Specified by:

getComponentConfiguration in interface org.apache.storm.topology.IComponent

Overrides:

getComponentConfiguration in class org.apache.storm.topology.base.BaseComponent

prepare

public void prepare(Map<String,Object> stormConf,
                    org.apache.storm.task.TopologyContext context,
                    org.apache.storm.task.OutputCollector collector)

Specified by:

prepare in interface org.apache.storm.task.IBolt

Overrides:

prepare in class StatusEmitterBolt

declareOutputFields

public void declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)

Specified by:

declareOutputFields in interface org.apache.storm.topology.IComponent

Overrides:

declareOutputFields in class StatusEmitterBolt

cleanup

public void cleanup()

Specified by:

cleanup in interface org.apache.storm.task.IBolt

Overrides:

cleanup in class org.apache.storm.topology.base.BaseRichBolt

execute

public void execute(org.apache.storm.tuple.Tuple input)