Package com.cedarsoftware.util

Class UrlUtilities

  • public final class UrlUtilities
extends Object
    Useful utilities for working with UrlConnections and IO. Anyone using the deprecated api calls for proxying to urls should update to use the new suggested calls. To let the jvm proxy for you automatically, use the following -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg
    Author:
    Ken Partlow, John DeRegnaucourt ([email protected])
    Copyright (c) Cedar Software LLC

    Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

    • Method Detail

      • clearGlobalUserAgent

        public static void clearGlobalUserAgent()

      • clearGlobalReferrer

        public static void clearGlobalReferrer()

      • setReferrer

        public static void setReferrer​(String referer)

      • getReferrer

        public static String getReferrer()

      • setUserAgent

        public static void setUserAgent​(String agent)

      • getUserAgent

        public static String getUserAgent()

      • readErrorResponse

        public static void readErrorResponse​(URLConnection c)

      • getCookies

        public static void getCookies​(URLConnection conn,
                              Map store)
        Retrieves and stores cookies returned by the host on the other side of the open java.net.URLConnection.

        The connection MUST have been opened using the connect() method or a IOException will be thrown.

        Parameters:
        conn - a java.net.URLConnection - must be open, or IOException will be thrown

      • setCookies

        public static void setCookies​(URLConnection conn,
                              Map store)
                       throws IOException
        Prior to opening a URLConnection, calling this method will set all unexpired cookies that match the path or subpaths for thi underlying URL

        The connection MUST NOT have been opened method or an IOException will be thrown.

        Parameters:
        conn - a java.net.URLConnection - must NOT be open, or IOException will be thrown
        Throws:
        IOException - Thrown if conn has already been opened.

      • getCookieDomainFromHost

        public static String getCookieDomainFromHost​(String host)

      • getContentFromUrlAsString

        public static String getContentFromUrlAsString​(String url)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.
        Parameters:
        url - URL to hit
        Returns:
        UTF-8 String read from URL or null in the case of error.

      • getContentFromUrlAsString

        public static String getContentFromUrlAsString​(URL url,
                                               boolean allowAllCerts)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.
        Parameters:
        url - URL to hit
        allowAllCerts - true to not verify certificates
        Returns:
        UTF-8 String read from URL or null in the case of error.

      • getContentFromUrlAsString

        public static String getContentFromUrlAsString​(String url,
                                               Map inCookies,
                                               Map outCookies,
                                               boolean trustAllCerts)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.
        Parameters:
        url - URL to hit
        inCookies - Map of session cookies (or null if not needed)
        outCookies - Map of session cookies (or null if not needed)
        trustAllCerts - if true, SSL connection will always be trusted.
        Returns:
        String of content fetched from URL.

      • getContentFromUrlAsString

        public static String getContentFromUrlAsString​(URL url,
                                               Map inCookies,
                                               Map outCookies,
                                               boolean trustAllCerts)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String.
        Parameters:
        url - URL to hit
        inCookies - Map of session cookies (or null if not needed)
        outCookies - Map of session cookies (or null if not needed)
        trustAllCerts - if true, SSL connection will always be trusted.
        Returns:
        String of content fetched from URL.

      • getContentFromUrl

        public static byte[] getContentFromUrl​(String url)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].
        Parameters:
        url - URL to hit
        Returns:
        byte[] read from URL or null in the case of error.

      • getContentFromUrl

        public static byte[] getContentFromUrl​(URL url,
                                       boolean allowAllCerts)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].
        Parameters:
        url - URL to hit
        Returns:
        byte[] read from URL or null in the case of error.

      • getContentFromUrl

        public static byte[] getContentFromUrl​(String url,
                                       Map inCookies,
                                       Map outCookies,
                                       boolean allowAllCerts)

      • getContentFromUrl

        public static byte[] getContentFromUrl​(URL url,
                                       Map inCookies,
                                       Map outCookies,
                                       boolean allowAllCerts)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].
        Parameters:
        url - URL to hit
        inCookies - Map of session cookies (or null if not needed)
        outCookies - Map of session cookies (or null if not needed)
        allowAllCerts - override certificate validation?
        Returns:
        byte[] of content fetched from URL.

      • getContentFromUrl

        public static byte[] getContentFromUrl​(String url,
                                       Map inCookies,
                                       Map outCookies)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].
        Parameters:
        url - URL to hit
        inCookies - Map of session cookies (or null if not needed)
        outCookies - Map of session cookies (or null if not needed)
        Returns:
        byte[] of content fetched from URL.

      • getConnection

        public static URLConnection getConnection​(String url,
                                          boolean input,
                                          boolean output,
                                          boolean cache)
                                   throws IOException
        Parameters:
        input - boolean indicating whether this connection will be used for input
        output - boolean indicating whether this connection will be used for output
        cache - boolean allow caching (be careful setting this to true for non-static retrievals).
        Returns:
        URLConnection established URL connection.
        Throws:
        IOException

      • getConnection

        public static URLConnection getConnection​(URL url,
                                          boolean input,
                                          boolean output,
                                          boolean cache)
                                   throws IOException
        Parameters:
        input - boolean indicating whether this connection will be used for input
        output - boolean indicating whether this connection will be used for output
        cache - boolean allow caching (be careful setting this to true for non-static retrievals).
        Returns:
        URLConnection established URL connection.
        Throws:
        IOException

      • getConnection

        public static URLConnection getConnection​(URL url,
                                          Map inCookies,
                                          boolean input,
                                          boolean output,
                                          boolean cache,
                                          boolean allowAllCerts)
                                   throws IOException
        Gets a connection from a url. All getConnection calls should go through this code.
        Parameters:
        inCookies - Supply cookie Map (received from prior setCookies calls from server)
        input - boolean indicating whether this connection will be used for input
        output - boolean indicating whether this connection will be used for output
        cache - boolean allow caching (be careful setting this to true for non-static retrievals).
        Returns:
        URLConnection established URL connection.
        Throws:
        IOException

      • getConnection

        @Deprecated
public static URLConnection getConnection​(URL url,
                                          Map inCookies,
                                          boolean input,
                                          boolean output,
                                          boolean cache,
                                          Proxy proxy,
                                          boolean allowAllCerts)
                                   throws IOException
        Deprecated.
        As of release 1.13.0, replaced by getConnection(java.net.URL, java.util.Map, boolean, boolean, boolean, boolean)
        Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg
        Throws:
        IOException

      • getConnection

        @Deprecated
public static URLConnection getConnection​(URL url,
                                          String server,
                                          int port,
                                          Map inCookies,
                                          boolean input,
                                          boolean output,
                                          boolean cache,
                                          boolean allowAllCerts)
                                   throws IOException
        Deprecated.
        As of release 1.13.0, replaced by getConnection(java.net.URL, java.util.Map, boolean, boolean, boolean, boolean)
        Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg
        Throws:
        IOException

      • getContentFromUrl

        @Deprecated
public static byte[] getContentFromUrl​(String url,
                                       Proxy proxy)
        Deprecated.
        As of release 1.13.0, replaced by getContentFromUrl(String)
        Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].
        Parameters:
        url - URL to hit
        proxy - proxy to use to create connection
        Returns:
        byte[] read from URL or null in the case of error.

      • getContentFromUrl

        @Deprecated
public static byte[] getContentFromUrl​(String url,
                                       Proxy proxy,
                                       SSLSocketFactory factory,
                                       HostnameVerifier verifier)
        Deprecated.
        As of release 1.13.0, replaced by getContentFromUrl(String)
        Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[].
        Parameters:
        url - URL to hit
        proxy - Proxy server to create connection (or null if not needed)
        factory - custom SSLSocket factory (or null if not needed)
        verifier - custom Hostnameverifier (or null if not needed)
        Returns:
        byte[] of content fetched from URL.

      • getContentFromUrl

        @Deprecated
public static byte[] getContentFromUrl​(String url,
                                       Map inCookies,
                                       Map outCookies,
                                       Proxy proxy,
                                       SSLSocketFactory factory,
                                       HostnameVerifier verifier)
        Deprecated.
        As of release 1.13.0, replaced by getContentFromUrl(String)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[]. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg
        Parameters:
        url - URL to hit
        proxy - Proxy server to create connection (or null if not needed)
        factory - custom SSLSocket factory (or null if not needed)
        verifier - custom Hostnameverifier (or null if not needed)
        Returns:
        byte[] of content fetched from URL.

      • getContentFromUrlAsString

        @Deprecated
public static String getContentFromUrlAsString​(String url,
                                               Proxy proxy)
        Deprecated.
        As of release 1.13.0, replaced by getContentFromUrl(String)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[]. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg
        Parameters:
        url - URL to hit
        proxy - proxy to use to create connection
        Returns:
        String read from URL or null in the case of error.

      • getContentFromUrl

        @Deprecated
public static byte[] getContentFromUrl​(URL url,
                                       Map inCookies,
                                       Map outCookies,
                                       Proxy proxy,
                                       boolean allowAllCerts)
        Deprecated.
        As of release 1.13.0, replaced by getContentFromUrl(String, java.util.Map, java.util.Map, boolean)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[]. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg
        Parameters:
        url - URL to hit
        inCookies - Map of session cookies (or null if not needed)
        outCookies - Map of session cookies (or null if not needed)
        proxy - Proxy server to create connection (or null if not needed)
        Returns:
        byte[] of content fetched from URL.

      • getContentFromUrl

        @Deprecated
public static byte[] getContentFromUrl​(String url,
                                       Map inCookies,
                                       Map outCookies,
                                       Proxy proxy,
                                       boolean allowAllCerts)
        Deprecated.
        As of release 1.13.0, replaced by getConnection(String, boolean, boolean, boolean)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[]. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg
        Parameters:
        url - URL to hit
        inCookies - Map of session cookies (or null if not needed)
        outCookies - Map of session cookies (or null if not needed)
        proxy - Proxy server to create connection (or null if not needed)
        Returns:
        byte[] of content fetched from URL.

      • getContentFromUrl

        @Deprecated
public static byte[] getContentFromUrl​(String url,
                                       String proxyServer,
                                       int port,
                                       Map inCookies,
                                       Map outCookies,
                                       boolean allowAllCerts)
        Deprecated.
        As of release 1.13.0, replaced by getContentFromUrl(String, java.util.Map, java.util.Map, boolean)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a byte[]. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) http.nonProxyHosts (should always include localhost) https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg
        Parameters:
        url - URL to hit
        proxyServer - String named of proxy server
        port - port to access proxy server
        inCookies - Map of session cookies (or null if not needed)
        outCookies - Map of session cookies (or null if not needed)
        allowAllCerts - if true, SSL connection will always be trusted.
        Returns:
        byte[] of content fetched from URL.

      • getContentFromUrlAsString

        @Deprecated
public static String getContentFromUrlAsString​(String url,
                                               String proxyServer,
                                               int port,
                                               Map inCookies,
                                               Map outCookies,
                                               boolean ignoreSec)
        Deprecated.
        As of release 1.13.0, replaced by getContentFromUrlAsString(String, java.util.Map, java.util.Map, boolean)
        Get content from the passed in URL. This code will open a connection to the passed in server, fetch the requested content, and return it as a String. Anyone using the proxy calls such as this one should have that managed by the jvm with -D parameters: http.proxyHost http.proxyPort (default: 80) always * https.proxyHost https.proxyPort Example: -Dhttp.proxyHost=proxy.example.org -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.org -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=*.foo.com|localhost|*.td.afg
        Parameters:
        url - URL to hit
        proxyServer - String named of proxy server
        port - port to access proxy server
        inCookies - Map of session cookies (or null if not needed)
        outCookies - Map of session cookies (or null if not needed)
        ignoreSec - if true, SSL connection will always be trusted.
        Returns:
        String of content fetched from URL.