Package com.digitalpebble.stormcrawler.util

Class URLPartitioner

java.lang.Object

com.digitalpebble.stormcrawler.util.URLPartitioner

public class URLPartitioner
extends Object

Generates a partition key for a given URL based on the hostname, domain or IP address. This can be called by the URLPartitionerBolt or any other component.

Constructor Summary

Constructors

Constructor	Description
URLPartitioner()

Method Summary

Modifier and Type	Method	Description
void	configure(Map stormConf)
String	getPartition(String url, Metadata metadata)	Returns the host, domain, IP of a URL so that it can be partitioned for politeness, depending on the value of the config partition.url.mode.
static String	getPartition(String url, Metadata metadata, String partitionMode)	Returns the host, domain, IP of a URL so that it can be partitioned for politeness, depending on the value of the parameter partitionMode.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

URLPartitioner

public URLPartitioner()

Method Detail

getPartition

public static String getPartition(String url,
                                  Metadata metadata,
                                  String partitionMode)

Returns the host, domain, IP of a URL so that it can be partitioned for politeness, depending on the value of the parameter partitionMode.

getPartition

public String getPartition(String url,
                           Metadata metadata)

Returns the host, domain, IP of a URL so that it can be partitioned for politeness, depending on the value of the config partition.url.mode.

configure

public void configure(Map stormConf)