Class PriorityURLBuffer
- java.lang.Object
-
- com.digitalpebble.stormcrawler.persistence.urlbuffer.AbstractURLBuffer
-
- com.digitalpebble.stormcrawler.persistence.urlbuffer.SimpleURLBuffer
-
- com.digitalpebble.stormcrawler.persistence.urlbuffer.PriorityURLBuffer
-
- All Implemented Interfaces:
URLBuffer
public class PriorityURLBuffer extends SimpleURLBuffer
Determines the priority of the buffers based on the number of URLs acked in a configurable period of time.- Since:
- 1.16
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class com.digitalpebble.stormcrawler.persistence.urlbuffer.AbstractURLBuffer
AbstractURLBuffer.URLMetadata
-
-
Field Summary
-
Fields inherited from class com.digitalpebble.stormcrawler.persistence.urlbuffer.AbstractURLBuffer
in_buffer, listener, partitioner, queues
-
Fields inherited from interface com.digitalpebble.stormcrawler.persistence.urlbuffer.URLBuffer
bufferClassParamName
-
-
Constructor Summary
Constructors Constructor Description PriorityURLBuffer()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
acked(String url)
Notify the buffer that a URL has been successfully processed used e.g to compute an ideal delay for a host queueorg.apache.storm.tuple.Values
next()
Retrieves the next available URL, guarantees that the URLs are always perfectly shuffled-
Methods inherited from class com.digitalpebble.stormcrawler.persistence.urlbuffer.AbstractURLBuffer
add, add, configure, hasNext, numQueues, setEmptyQueueListener, size
-
-
-
-
Method Detail
-
next
public org.apache.storm.tuple.Values next()
Description copied from class:SimpleURLBuffer
Retrieves the next available URL, guarantees that the URLs are always perfectly shuffled- Specified by:
next
in interfaceURLBuffer
- Overrides:
next
in classSimpleURLBuffer
- Returns:
- null if no entries are available
-
-