uk.ac.gate.cloud.data

Class DataManager

java.lang.Object

uk.ac.gate.cloud.data.DataManager

public class DataManager
extends Object

Main entry point for the data bundle API to create, manage, and download data bundles that you own.

Author:

Ian Roberts

Constructor Summary

Constructors

Constructor and Description
DataManager(RestClient client) Construct a DataManager using the given RestClient to communicate with the API.
DataManager(String apiKeyId, String apiPassword) Construct a DataManager accessing the GATE Cloud public API with the given credentials.

Method Summary

Modifier and Type	Method and Description
DataBundle	createARCBundleFromS3(String bundleName, String accessKeyId, String secretKey, InputType inputType, String encoding, String mimeTypeOverride, String mimeTypes, String... locations) Create a new data bundle from a list of s3://bucket/key URLs that point to ARC or WARC archives that are already hosted on Amazon S3.
DataBundle	createARCBundleFromUploads(String bundleName, InputType inputType, String encoding, String mimeTypeOverride, String mimeTypes, File... localFiles) Create a new data bundle by uploading local ARC or WARC archives to GATE Cloud managed storage.
DataBundle	createArchiveBundleFromS3(String bundleName, String accessKeyId, String secretKey, InputType inputType, String encoding, String mimeTypeOverride, String fileExtensions, String... locations) Create a new data bundle from a list of s3://bucket/key URLs that point to ZIP or TAR archives or Twitter JSON files that are already hosted on Amazon S3.
DataBundle	createArchiveBundleFromUploads(String bundleName, InputType inputType, String encoding, String mimeTypeOverride, String fileExtensions, File... localFiles) Create a new data bundle by uploading local ZIP or TAR archives or Twitter JSON files to GATE Cloud managed storage.
protected DataBundle	createS3Bundle(String bundleName, String accessKeyId, String secretKey, InputType inputType, String encoding, String mimeTypeOverride, String fileExtensions, String mimeTypes, String... locations) Common logic for creating bundles from S3.
DataBundle	getBundle(long id) Get details of a particular bundle given its ID.
DataBundle	getBundle(String url) Get details of a specific bundle given its detail URL (which will have been received from an earlier API call).
List<DataBundleSummary>	listBundles() List all the data bundles that are owned by the authenticating user
protected DataBundle	uploadBundle(String bundleName, InputType inputType, String encoding, String mimeTypeOverride, String fileExtensions, String mimeTypes, File... localFiles) Common logic for creating bundles from uploads.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataManager

public DataManager(RestClient client)

Construct a DataManager using the given RestClient to communicate with the API.

Parameters:

client - the client object used for communication

DataManager

public DataManager(String apiKeyId,
                   String apiPassword)

Construct a DataManager accessing the GATE Cloud public API with the given credentials.

Parameters:

apiKeyId - API key ID for authentication

apiPassword - corresponding password

Method Detail

listBundles

public List<DataBundleSummary> listBundles()

List all the data bundles that are owned by the authenticating user

Returns:

list of DataBundleSummary objects representing the user's bundles - call the details method to get the full detail (which requires another API call).

getBundle

public DataBundle getBundle(long id)

Get details of a particular bundle given its ID.

Parameters:

id - the ID of the required bundle

Returns:

the bundle details

getBundle

public DataBundle getBundle(String url)

Get details of a specific bundle given its detail URL (which will have been received from an earlier API call).

Parameters:

url - the detail URL for the required bundle

Returns:

the bundle details

createArchiveBundleFromS3

public DataBundle createArchiveBundleFromS3(String bundleName,
                                            String accessKeyId,
                                            String secretKey,
                                            InputType inputType,
                                            String encoding,
                                            String mimeTypeOverride,
                                            String fileExtensions,
                                            String... locations)

Create a new data bundle from a list of s3://bucket/key URLs that point to ZIP or TAR archives or Twitter JSON files that are already hosted on Amazon S3. Note that all the files in a bundle must be the same type - one bundle cannot contain a mixture of ZIP and TAR archives, for example. Job parameters for the file encoding, MIME type, and the file extensions used to filter the entries from the archives can also be supplied, and will feed through to any annotation jobs that take their input from the created bundle.

Parameters:

bundleName - a name for the new bundle.

accessKeyId - an AWS access key ID (typically a limited IAM user) with permission to get the specified object

secretKey - the corresponding AWS secret key.

inputType - the type of the input

encoding - character encoding to use when reading entries from the archive. If null, UTF-8 will be used. Should be left as null for Twitter input types.

mimeTypeOverride - the MIME type to use when parsing entries from the archive. If null the appropriate type will be guessed based on the file name extension. Should be left as null for Twitter input types.

fileExtensions - comma-separated list of file extensions that will be processed. Entries that do not match any of these extensions will be ignored. If null all entries that represent files (as opposed to directories) will be processed. Should be left as null for Twitter input types.

locations - "URLs" of the form s3://bucketname/key denoting the target objects in Amazon S3

Returns:

details of the newly created bundle.

createARCBundleFromS3

public DataBundle createARCBundleFromS3(String bundleName,
                                        String accessKeyId,
                                        String secretKey,
                                        InputType inputType,
                                        String encoding,
                                        String mimeTypeOverride,
                                        String mimeTypes,
                                        String... locations)

Create a new data bundle from a list of s3://bucket/key URLs that point to ARC or WARC archives that are already hosted on Amazon S3. Note that all the files in a bundle must be the same type - one bundle cannot contain a mixture of ARC and WARC archives, for example. Job parameters for the file encoding, MIME type, and the MIME types used to filter the entries from the archives can also be supplied, and will feed through to any annotation jobs that take their input from the created bundle.

Parameters:

bundleName - a name for the new bundle.

accessKeyId - an AWS access key ID (typically a limited IAM user) with permission to get the specified object

secretKey - the corresponding AWS secret key.

inputType - the type of the input

encoding - character encoding to use when reading entries from the archive. If null, UTF-8 will be used. Should be left as null for Twitter input types.

mimeTypeOverride - the MIME type to use when parsing entries from the archive. If null the appropriate type will be guessed based on the file name extension. Should be left as null for Twitter input types.

mimeTypes - space-separated list of MIME types used to filter the entries of interest from the ARC file. Entries whose MIME type does not match any of these will be ignored.

locations - "URLs" of the form s3://bucketname/key denoting the target objects in Amazon S3

Returns:

details of the newly created bundle.

createS3Bundle

protected DataBundle createS3Bundle(String bundleName,
                                    String accessKeyId,
                                    String secretKey,
                                    InputType inputType,
                                    String encoding,
                                    String mimeTypeOverride,
                                    String fileExtensions,
                                    String mimeTypes,
                                    String... locations)

Common logic for creating bundles from S3.

createArchiveBundleFromUploads

public DataBundle createArchiveBundleFromUploads(String bundleName,
                                                 InputType inputType,
                                                 String encoding,
                                                 String mimeTypeOverride,
                                                 String fileExtensions,
                                                 File... localFiles)

Create a new data bundle by uploading local ZIP or TAR archives or Twitter JSON files to GATE Cloud managed storage. Note that all the files in a bundle must be the same type - one bundle cannot contain a mixture of ZIP and TAR archives, for example. Job parameters for the file encoding, MIME type, and the file extensions used to filter the entries from the archives can also be supplied, and will feed through to any annotation jobs that take their input from the created bundle.

Parameters:

bundleName - a name for the new bundle.

inputType - the type of the input

encoding - character encoding to use when reading entries from the archive. If null, UTF-8 will be used. Should be left as null for Twitter input types.

mimeTypeOverride - the MIME type to use when parsing entries from the archive. If null the appropriate type will be guessed based on the file name extension. Should be left as null for Twitter input types.

fileExtensions - comma-separated list of file extensions that will be processed. Entries that do not match any of these extensions will be ignored. If null all entries that represent files (as opposed to directories) will be processed. Should be left as null for Twitter input types.

localFiles - the files to upload

Returns:

details of the newly created bundle.

createARCBundleFromUploads

public DataBundle createARCBundleFromUploads(String bundleName,
                                             InputType inputType,
                                             String encoding,
                                             String mimeTypeOverride,
                                             String mimeTypes,
                                             File... localFiles)

Create a new data bundle by uploading local ARC or WARC archives to GATE Cloud managed storage. Note that all the files in a bundle must be the same type - one bundle cannot contain a mixture of ARC and WARC archives, for example. Job parameters for the file encoding, MIME type, and the MIME types used to filter the entries from the archives can also be supplied, and will feed through to any annotation jobs that take their input from the created bundle.

Parameters:

bundleName - a name for the new bundle.

inputType - the type of the input

encoding - character encoding to use when reading entries from the archive. If null, UTF-8 will be used. Should be left as null for Twitter input types.

mimeTypeOverride - the MIME type to use when parsing entries from the archive. If null the appropriate type will be guessed based on the file name extension. Should be left as null for Twitter input types.

mimeTypes - space-separated list of MIME types used to filter the entries of interest from the ARC file. Entries whose MIME type does not match any of these will be ignored.

localFiles - the files to upload

Returns:

details of the newly created bundle.

uploadBundle

protected DataBundle uploadBundle(String bundleName,
                                  InputType inputType,
                                  String encoding,
                                  String mimeTypeOverride,
                                  String fileExtensions,
                                  String mimeTypes,
                                  File... localFiles)

Common logic for creating bundles from uploads.