Package com.digitalpebble.stormcrawler.bolt

Class SimpleFetcherBolt

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichBolt

com.digitalpebble.stormcrawler.bolt.StatusEmitterBolt

com.digitalpebble.stormcrawler.bolt.SimpleFetcherBolt

All Implemented Interfaces:

Serializable, org.apache.storm.task.IBolt, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichBolt

public class SimpleFetcherBolt
extends StatusEmitterBolt

A simple fetcher with no internal queues. This bolt either enforces the delay set by the configuration or robots.txt by either sleeping or resending the tuple to itself on the THROTTLE_STREAM using Direct grouping.

 .directGrouping("fetch", "throttle")
 

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
static String	QUEUE_MODE_DOMAIN
static String	QUEUE_MODE_HOST
static String	QUEUE_MODE_IP
static String	THROTTLE_STREAM

Fields inherited from class com.digitalpebble.stormcrawler.bolt.StatusEmitterBolt

collector

Constructor Summary

Constructors

Constructor	Description
SimpleFetcherBolt()

Method Summary

Modifier and Type	Method	Description
void	cleanup()
void	declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)
void	execute(org.apache.storm.tuple.Tuple input)
void	prepare(Map<String,Object> stormConf, org.apache.storm.task.TopologyContext context, org.apache.storm.task.OutputCollector collector)

Methods inherited from class com.digitalpebble.stormcrawler.bolt.StatusEmitterBolt

allowRedirs, emitOutlink, filterOutlink

Methods inherited from class org.apache.storm.topology.base.BaseComponent

getComponentConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.storm.topology.IComponent

getComponentConfiguration

Field Detail

QUEUE_MODE_HOST

public static final String QUEUE_MODE_HOST

See Also:

Constant Field Values

QUEUE_MODE_DOMAIN

public static final String QUEUE_MODE_DOMAIN

See Also:

Constant Field Values

QUEUE_MODE_IP

public static final String QUEUE_MODE_IP

See Also:

Constant Field Values

THROTTLE_STREAM

public static final String THROTTLE_STREAM

See Also:

Constant Field Values

Constructor Detail

SimpleFetcherBolt

public SimpleFetcherBolt()

Method Detail

prepare

public void prepare(Map<String,Object> stormConf,
                    org.apache.storm.task.TopologyContext context,
                    org.apache.storm.task.OutputCollector collector)

Specified by:

prepare in interface org.apache.storm.task.IBolt

Overrides:

prepare in class StatusEmitterBolt

declareOutputFields

public void declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)

Specified by:

declareOutputFields in interface org.apache.storm.topology.IComponent

Overrides:

declareOutputFields in class StatusEmitterBolt

cleanup

public void cleanup()

Specified by:

cleanup in interface org.apache.storm.task.IBolt

Overrides:

cleanup in class org.apache.storm.topology.base.BaseRichBolt

execute

public void execute(org.apache.storm.tuple.Tuple input)