Package com.digitalpebble.stormcrawler.persistence.urlbuffer

Class AbstractURLBuffer

java.lang.Object

com.digitalpebble.stormcrawler.persistence.urlbuffer.AbstractURLBuffer

All Implemented Interfaces:

URLBuffer

Direct Known Subclasses:

SchedulingURLBuffer, SimpleURLBuffer

public abstract class AbstractURLBuffer
extends Object
implements URLBuffer

Abstract class for URLBuffer interface, meant to simplify the code of the implementations and provide some default methods

Since:

1.15

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected class	AbstractURLBuffer.URLMetadata

Field Summary

Fields

Modifier and Type	Field	Description
protected Set<String>	in_buffer
protected EmptyQueueListener	listener
protected URLPartitioner	partitioner
protected Map<String,Queue<AbstractURLBuffer.URLMetadata>>	queues

Fields inherited from interface com.digitalpebble.stormcrawler.persistence.urlbuffer.URLBuffer

bufferClassParamName

Constructor Summary

Constructors

Constructor	Description
AbstractURLBuffer()

Method Summary

Modifier and Type	Method	Description
boolean	add(String URL, Metadata m)	Stores the URL and its Metadata using the hostname as key.
boolean	add(String URL, Metadata m, String key)	Stores the URL and its Metadata under a given key.
void	configure(Map<String,Object> stormConf)
boolean	hasNext()	Implementations of this method should be synchronised
int	numQueues()	Total number of queues in the buffer *
void	setEmptyQueueListener(EmptyQueueListener l)
int	size()	Total number of URLs in the buffer *

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.digitalpebble.stormcrawler.persistence.urlbuffer.URLBuffer

acked, next

Field Detail

in_buffer

protected Set<String> in_buffer

listener

protected EmptyQueueListener listener

partitioner

protected final URLPartitioner partitioner

queues

protected final Map<String,Queue<AbstractURLBuffer.URLMetadata>> queues

Constructor Detail

AbstractURLBuffer

public AbstractURLBuffer()

Method Detail

configure

public void configure(Map<String,Object> stormConf)

Specified by:

configure in interface URLBuffer

numQueues

public int numQueues()

Total number of queues in the buffer *

Specified by:

numQueues in interface URLBuffer

add

public boolean add(String URL,
                   Metadata m,
                   String key)

Stores the URL and its Metadata under a given key.

Specified by:

add in interface URLBuffer

Returns:

false if the URL was already in the buffer, true if it wasn't and was added

add

public boolean add(String URL,
                   Metadata m)

Stores the URL and its Metadata using the hostname as key.

Specified by:

add in interface URLBuffer

Returns:

false if the URL was already in the buffer, true if it wasn't and was added

size

public int size()

Total number of URLs in the buffer *

Specified by:

size in interface URLBuffer

setEmptyQueueListener

public void setEmptyQueueListener(EmptyQueueListener l)

Specified by:

setEmptyQueueListener in interface URLBuffer

hasNext

public boolean hasNext()

Description copied from interface: URLBuffer

Implementations of this method should be synchronised

Specified by:

hasNext in interface URLBuffer