Package com.digitalpebble.stormcrawler.protocol.httpclient

Class HttpProtocol

java.lang.Object

com.digitalpebble.stormcrawler.protocol.AbstractHttpProtocol

com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol

All Implemented Interfaces:

Protocol, org.apache.http.client.ResponseHandler<ProtocolResponse>

public class HttpProtocol
extends AbstractHttpProtocol
implements org.apache.http.client.ResponseHandler<ProtocolResponse>

Uses Apache httpclient to handle http and https

Nested Class Summary

Nested classes/interfaces inherited from class com.digitalpebble.stormcrawler.protocol.AbstractHttpProtocol

AbstractHttpProtocol.KeyValue

Field Summary

Fields inherited from class com.digitalpebble.stormcrawler.protocol.AbstractHttpProtocol

customHeaders, protocolMDprefix, protocolVersions, proxyManager, RESPONSE_COOKIES_HEADER, SET_HEADER_BY_REQUEST, skipRobots, storeHTTPHeaders, useCookies

Constructor Summary

Constructors

Constructor	Description
HttpProtocol()

Method Summary

Modifier and Type	Method	Description
protected void	addHeadersToRequest(org.apache.http.client.methods.HttpRequestBase request, Metadata md)
void	configure(org.apache.storm.Config conf)
ProtocolResponse	getProtocolOutput(String url, Metadata md)	Fetches the content and additional metadata
ProtocolResponse	handleResponse(org.apache.http.HttpResponse response)
ProtocolResponse	handleResponseWithContentLimit(org.apache.http.HttpResponse response, int maxContent)
static void	main(String[] args)

Methods inherited from class com.digitalpebble.stormcrawler.protocol.AbstractHttpProtocol

cleanup, getAgentString, getRobotRules

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HttpProtocol

public HttpProtocol()

Method Detail

configure

public void configure(org.apache.storm.Config conf)

Specified by:

configure in interface Protocol

Overrides:

configure in class AbstractHttpProtocol

getProtocolOutput

public ProtocolResponse getProtocolOutput(String url,
                                          Metadata md)
                                   throws Exception

Description copied from interface: Protocol

Fetches the content and additional metadata

IMPORTANT: the metadata returned within the response should only be new additional, no need to return the metadata passed in.

Specified by:

getProtocolOutput in interface Protocol

Parameters:

url - the location of the content

md - extra information

Returns:

the content and optional metadata fetched via this protocol

Throws:

Exception

addHeadersToRequest

protected void addHeadersToRequest(org.apache.http.client.methods.HttpRequestBase request,
                                   Metadata md)

handleResponse

public ProtocolResponse handleResponse(org.apache.http.HttpResponse response)
                                throws IOException

Specified by:

handleResponse in interface org.apache.http.client.ResponseHandler<ProtocolResponse>

Throws:

IOException

handleResponseWithContentLimit

public ProtocolResponse handleResponseWithContentLimit(org.apache.http.HttpResponse response,
                                                       int maxContent)
                                                throws IOException

Throws:

IOException

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception