Package org.archive.modules.net

Class CrawlHost

java.lang.Object
org.archive.modules.net.CrawlHost
All Implemented Interfaces:
Serializable, FetchStats.HasFetchStats, org.archive.util.IdentityCacheable
public class CrawlHost
extends Object
implements Serializable, FetchStats.HasFetchStats, org.archive.util.IdentityCacheable
Represents a single remote "host". An host is a name for which there is a dns record or an IP-address. This might be a machine or a virtual host.
Author:
gojomo
See Also:
Serialized Form

  • Field Details

    • IP_NEVER_EXPIRES

      public static final long IP_NEVER_EXPIRES
      Flag value indicating always-valid IP
      See Also:
      Constant Field Values

    • IP_NEVER_LOOKED_UP

      public static final long IP_NEVER_LOOKED_UP
      Flag value indicating an IP has not yet been looked up
      See Also:
      Constant Field Values

    • substats

      protected FetchStats substats

  • Constructor Details

    • CrawlHost

      public CrawlHost​(String hostname)
      Create a new CrawlHost object.
      Parameters:
      hostname - the host name for this host.

    • CrawlHost

      public CrawlHost​(String hostname, String countryCode)
      Create a new CrawlHost object.
      Parameters:
      hostname - the host name for this host.
      countryCode - the country code for this host.

  • Method Details

    • hasBeenLookedUp

      public boolean hasBeenLookedUp()
      Return true if the IP for this host has been looked up. Returns true even if the lookup failed.
      Returns:
      true if the IP for this host has been looked up.

    • setIP

      public void setIP​(InetAddress address, long ttl)
      Set the IP address for this host.
      Parameters:
      address -
      ttl - the TTL from the dns record in seconds or -1 if it should live forever (is a numeric IP).

    • getIP

      public InetAddress getIP()
      Get the IP address for this host.
      Returns:
      the IP address for this host.

    • getIpFetched

      public long getIpFetched()
      Get the time when the IP address for this host was last looked up.
      Returns:
      the time when the IP address for this host was last looked up.

    • getIpTTL

      public long getIpTTL()
      Get the TTL value from the dns record for this host.
      Returns:
      the TTL value from the dns record for this host -- in seconds -- or -1 if this lookup should be valid forever (numeric ip).

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • equals

      public boolean equals​(Object obj)
      Overrides:
      equals in class Object

    • getHostName

      public String getHostName()
      Get the host name.
      Returns:
      Returns the host name.

    • getEarliestNextURIEmitTime

      public long getEarliestNextURIEmitTime()
      Get the earliest time a URI for this host could be emitted. This only has effect if constraints on bandwidth per host is set.
      Returns:
      Returns the earliestNextURIEmitTime.

    • setEarliestNextURIEmitTime

      public void setEarliestNextURIEmitTime​(long earliestNextURIEmitTime)
      Set the earliest time a URI for this host could be emitted. This only has effect if constraints on bandwidth per host is set.
      Parameters:
      earliestNextURIEmitTime - The earliestNextURIEmitTime to set.

    • getCountryCode

      public String getCountryCode()
      Get country code of this host
      Returns:
      Retruns country code or null if not availabe

    • setCountryCode

      public void setCountryCode​(String countryCode)
      Set country code for this hos
      Parameters:
      countryCode - The country code of this host

    • getSubstats

      public FetchStats getSubstats()
      Specified by:
      getSubstats in interface FetchStats.HasFetchStats

    • autoregisterTo

      public static void autoregisterTo​(org.archive.bdb.AutoKryo kryo)

    • getKey

      public String getKey()
      Specified by:
      getKey in interface org.archive.util.IdentityCacheable

    • makeDirty

      public void makeDirty()
      Specified by:
      makeDirty in interface org.archive.util.IdentityCacheable

    • setIdentityCache

      public void setIdentityCache​(org.archive.util.ObjectIdentityCache<?> cache)
      Specified by:
      setIdentityCache in interface org.archive.util.IdentityCacheable

    • fixUpName

      public String fixUpName()