Class MemorySpout
- java.lang.Object
-
- org.apache.storm.topology.base.BaseComponent
-
- org.apache.storm.topology.base.BaseRichSpout
-
- com.digitalpebble.stormcrawler.spout.MemorySpout
-
- All Implemented Interfaces:
Serializable
,org.apache.storm.spout.ISpout
,org.apache.storm.topology.IComponent
,org.apache.storm.topology.IRichSpout
public class MemorySpout extends org.apache.storm.topology.base.BaseRichSpout
Stores URLs in memory. Useful for testing and debugging in local mode or with a single worker. Uses StringTabScheme to parse the lines into URLs and Metadata, generates tuples on the default stream unless withDiscoveredStatus is set to true. Can be used with the MemoryStatusUpdater to receive discovered URLs and emulate a recursive crawl.- See Also:
- Serialized Form
-
-
Constructor Summary
Constructors Constructor Description MemorySpout(boolean withDiscoveredStatus, String... urls)
Emits tuples with DISCOVERED status, which is useful when injecting seeds directly to a statusupdaterbolt.MemorySpout(String... urls)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
activate()
static void
add(String url, Metadata md, Date nextFetch)
Add a new URL with the given metadata and nextFetch-datevoid
deactivate()
void
declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)
void
nextTuple()
void
open(Map<String,Object> conf, org.apache.storm.task.TopologyContext context, org.apache.storm.spout.SpoutOutputCollector collector)
-
-
-
Constructor Detail
-
MemorySpout
public MemorySpout(String... urls)
-
MemorySpout
public MemorySpout(boolean withDiscoveredStatus, String... urls)
Emits tuples with DISCOVERED status, which is useful when injecting seeds directly to a statusupdaterbolt.- Parameters:
withDiscoveredStatus
- whether the tuples generated should contain a Status field with DISCOVERED as value and be emitted on the status stream
-
-
Method Detail
-
add
public static void add(String url, Metadata md, Date nextFetch)
Add a new URL with the given metadata and nextFetch-date
-
open
public void open(Map<String,Object> conf, org.apache.storm.task.TopologyContext context, org.apache.storm.spout.SpoutOutputCollector collector)
-
nextTuple
public void nextTuple()
-
declareOutputFields
public void declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)
-
activate
public void activate()
- Specified by:
activate
in interfaceorg.apache.storm.spout.ISpout
- Overrides:
activate
in classorg.apache.storm.topology.base.BaseRichSpout
-
deactivate
public void deactivate()
- Specified by:
deactivate
in interfaceorg.apache.storm.spout.ISpout
- Overrides:
deactivate
in classorg.apache.storm.topology.base.BaseRichSpout
-
-