org.hpccsystems.ws.client

Class HPCCWsDFUClient

java.lang.Object

java.util.Observable

org.hpccsystems.ws.client.utils.DataSingleton

org.hpccsystems.ws.client.HPCCWsDFUClient

public class HPCCWsDFUClient
extends DataSingleton

Provides soap client for HPCC WsDFU web service.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATASET_ELEMENT
static String	ROW_ELEMENT
static String	WSDFUURI

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HPCCWsDFUClient(Connection baseConnection)
protected	HPCCWsDFUClient(String protocol, String targetHost, String targetPort, String user, String pass)
protected	HPCCWsDFUClient(WsDfuServiceSoapProxy wsDfuServiceSoapProxy)

Method Summary

Modifier and Type	Method and Description
DFUCreateFileWrapper	createFile(String fileName, String cluster, String eclRecordDefinition, int expirySeconds) Create a new (unpublished), uncompressed dfu file.
DFUCreateFileWrapper	createFile(String fileName, String cluster, String eclRecordDefinition, int expirySeconds, Boolean compressed) Create a new (unpublished) dfu file.
DFUCreateFileWrapper	createFile(String fileName, String cluster, String eclRecordDefinition, int expirySeconds, Boolean compressed, DFUFileTypeWrapper filetype, String requestid) Create a new (unpublished) dfu file.
DFUCreateFileWrapper	createFile(String fileName, String cluster, String eclRecordDefinition, String[] partitionHostMap, int expirySeconds) Create a new (unpublished) dfu file.
DFUCreateFileWrapper	createFileAndAcquireAccess(String fileName, String cluster, String eclRecordDefinition, int expirySeconds, Boolean compressed, DFUFileTypeWrapper type, String requestId) Create a new (unpublished) dfu file.
DFUCreateFileWrapper	createFileAndAcquireAccess(String fileName, String cluster, String eclRecordDefinition, String[] partitionHostMap, int expirySeconds, Boolean returnBinTypeInfo, Boolean returnJsonTypeInfo, FileAccessRole accessRole, SecAccessType accessType) Create a new (unpublished) dfu file.
List<DFUResult>	deleteFiles(Set<String> files, String cluster)
boolean	equals(Object aThat)
protected void	fastRefresh()
protected void	fullRefresh()
static HPCCWsDFUClient	get(Connection connection)
EclRecordInfo	getDatasetFields(String datasetname, String clusterName, String fieldSeparator) This method retrieves a list of columns for a THOR or CSV dataset.
DFUSearchDataResponse	getDFUData(String openLogicalName, String cluster, boolean roxieSelections, int chooseFile, int count, boolean schemaOnly, long startIndex) Return raw response from WsDFU DFUSearchData method requests
DFUFileAccessInfoWrapper	getFileAccess(SecAccessType accesstype, String filename, String clustername, int expiryseconds, String jobid, boolean includejsonTypeInfo, boolean includebinTypeInfo, boolean requestfileinfo) To be used in conjunction with DAFILESRV's rowservice distributed file read stream.
DFUFileAccessInfoWrapper	getFileAccess(String filename, String clustername, int expiryseconds, String jobid) To be used in conjunction with DAFILESRV's rowservice distributed file read/write stream.
String	getFileAccessBlob(SecAccessType accesstype, String filename, String clustername, int expiryseconds, String jobid) getFileAccessBlob - HPCC 7.0.x version The response is to be used in conjunction with DAFILESRV's rowservice distributed file read stream.
String	getFileAccessBlob(String filename, String clustername, int expiryseconds, String jobid) getFileAccessBlob The response is to be used in conjunction with DAFILESRV's rowservice distributed file read/write stream.
NodeList	getFileData(String logicalname, Long beginrow, Integer numrows, String clustername)
List<DFUDataColumnInfo>	getFileDataColumns(String logicalname, String clustername)
DFUFileDetailInfo	getFileDetails(String logicalname, String clustername)
DFUFileDetailInfo	getFileDetails(String logicalname, String clustername, boolean jsonTypeInfo, boolean binTypeInfo)
DFUInfoResponse	getFileInfo(String logicalname, String clustername)
DFUInfoResponse	getFileInfo(String logicalname, String clustername, boolean jsonTypeInfo, boolean binTypeInfo)
DFUDataColumn[]	getFileMetaData(String logicalname, String clustername) Deprecated.
List<DFUDataColumnInfo>	getFileMetaDataInfo(String logicalname, String clustername) Use this function to retrieve file metadata such as column information, for superfiles the metadata from the first subfile will be returned.
List<DFULogicalFileInfo>	getFiles(String scope)
String	getFirstRow(String datasetname, String clustername) Returns the first row of data for a dataset
String	getInitError() Returns error message encountered during initialization of wsdfuclient.
List<DFULogicalFileInfo>	getLogicalFiles(String filename, String cluster, int firstN, int pageStartFrom, int pageSize) Get array of logical files on target HPCC system based on input parameters
static int	getOriginalPort()
static URL	getOriginalURL()
static String	getOriginalWSDLURL() Provides the WSDL URL originally used to create the underlying stub code
WsDfuServiceSoapProxy	getSoapProxy() Provides soapproxy object for HPCCWsDFUClient which can be used to access the web service methods directly
int	hashCode()
boolean	hasInitError() Should be called after instantiation to confirm Successful initialization.
protected boolean	isComplete()
void	publishFile(String fileId, String eclRecordDefinition, long totalRecords, long fileSize) Publishes file created by WsDFUCreateFile.
void	publishFile(String fileId, String eclRecordDefinition, long totalRecords, long fileSize, Boolean overwrite) Publishes file created by WsDFUCreateFile.
List<DFULogicalFileInfo>	searchFiles(String logicalFilename, String cluster) searchFiles
void	setVerbose(boolean verbose)

Methods inherited from class org.hpccsystems.ws.client.utils.DataSingleton

monitor

Methods inherited from class java.util.Observable

addObserver, clearChanged, countObservers, deleteObserver, deleteObservers, hasChanged, notifyObservers, notifyObservers, setChanged

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

WSDFUURI

public static final String WSDFUURI

See Also:

Constant Field Values

ROW_ELEMENT

public static final String ROW_ELEMENT

See Also:

Constant Field Values

DATASET_ELEMENT

public static final String DATASET_ELEMENT

See Also:

Constant Field Values

Constructor Detail

HPCCWsDFUClient

protected HPCCWsDFUClient(WsDfuServiceSoapProxy wsDfuServiceSoapProxy)

Parameters:

wsDfuServiceSoapProxy -

HPCCWsDFUClient

protected HPCCWsDFUClient(Connection baseConnection)

Parameters:

baseConnection -

HPCCWsDFUClient

protected HPCCWsDFUClient(String protocol,
                          String targetHost,
                          String targetPort,
                          String user,
                          String pass)

Parameters:

protocol - - http or https

targetHost - server IP/name of the HPCC Cluster

targetPort - - port of the HPCC Cluster

user - - username to use when connecting to the HPCC Cluster

pass - - Password to use when connecting to the HPCC Cluster

Method Detail

getOriginalURL

public static URL getOriginalURL()
                          throws MalformedURLException

Throws:

MalformedURLException

getOriginalPort

public static int getOriginalPort()
                           throws MalformedURLException

Throws:

MalformedURLException

get

public static HPCCWsDFUClient get(Connection connection)

hasInitError

public boolean hasInitError()

Should be called after instantiation to confirm Successful initialization. The client init can fail due to many different types of issues including invalid connectivity options, invalid credentials, etc

Returns:

getInitError

public String getInitError()

Returns error message encountered during initialization of wsdfuclient. Empty string if no error encountered

Returns:

setVerbose

public void setVerbose(boolean verbose)

Parameters:

verbose - - sets verbose mode

getSoapProxy

public WsDfuServiceSoapProxy getSoapProxy()
                                   throws Exception

Provides soapproxy object for HPCCWsDFUClient which can be used to access the web service methods directly

Returns:

soapproxy for HPCCWsDFUClient

Throws:

Exception

getFileInfo

public DFUInfoResponse getFileInfo(String logicalname,
                                   String clustername)
                            throws Exception

Parameters:

logicalname - - logical file to get file info for, can start with '~' or not

clustername - - Optional. If specified, the cluster on which to search for the file

Returns:

a DFUInfoResponse object containing the file info

Throws:

Exception

getFileInfo

public DFUInfoResponse getFileInfo(String logicalname,
                                   String clustername,
                                   boolean jsonTypeInfo,
                                   boolean binTypeInfo)
                            throws Exception

Parameters:

logicalname - logical file name, can start with ~

clustername - optional, if specified the cluster name used in the search

jsonTypeInfo - want record structure information returned as JSON

binTypeInfo - want record structure information returned in binary format

Returns:

Throws:

Exception

getFileData

public NodeList getFileData(String logicalname,
                            Long beginrow,
                            Integer numrows,
                            String clustername)
                     throws Exception

Parameters:

logicalname - - logical file to retrieve data for

beginrow - - beginning row to retrieve data from

numrows - - number of rows of data to retrieve

clustername - - Optional. If specified, the cluster on which to find the logical file.

Returns:

an XML Element object holding the '' elements containing data.

Throws:

Exception

getFiles

public List<DFULogicalFileInfo> getFiles(String scope)
                                  throws Exception

Parameters:

scope - - file scope/directory to return files for

Returns:

an array of DFULogicalFile objects

Throws:

Exception

getFileMetaDataInfo

public List<DFUDataColumnInfo> getFileMetaDataInfo(String logicalname,
                                                   String clustername)
                                            throws Exception

Use this function to retrieve file metadata such as column information, for superfiles the metadata from the first subfile will be returned.

Parameters:

logicalname - - Logical filename.

clustername - - Optional - The cluster the logical filename is associated with.

Returns:

ArrayList of DFUDataColumnInfo

Throws:

Exception

getFileMetaData

@Deprecated
public DFUDataColumn[] getFileMetaData(String logicalname,
                                                   String clustername)
                                            throws Exception

Deprecated.

Deprecated, use getFileMetaDataInfo() Use this function to retrieve file metadata such as column information

Parameters:

logicalname - - Logical filename.

clustername - - Optional - The cluster the logical filename is associated with.

Returns:

Array of DFUDataColumns

Throws:

Exception

getFileDataColumns

public List<DFUDataColumnInfo> getFileDataColumns(String logicalname,
                                                  String clustername)
                                           throws Exception

Parameters:

logicalname - - logical filename to retrieve the dfu data columns for. Currently this method/service call functions for THOR files but will return nothing for CSV/XML/FLAT data files

clustername - - optional. The cluster the logical filename is associated with.

Returns:

ArrayList of DFUDataColumns

Throws:

Exception

getOriginalWSDLURL

public static String getOriginalWSDLURL()

Provides the WSDL URL originally used to create the underlying stub code

Returns:

original WSLD URL

getDatasetFields

public EclRecordInfo getDatasetFields(String datasetname,
                                      String clusterName,
                                      String fieldSeparator)
                               throws Exception

This method retrieves a list of columns for a THOR or CSV dataset. -if THOR, this is a straightforward service call to get the columns. -if CSV, depending on whether or not the CSV was sprayed or output by thor, it may or may not - have a record definition associated with it. If it doesn't, the number of fields in the CSV - is calculated by retrieving the first line and splitting it on the separator, and returning - a list of string Field1, Field2, etc. fields (that match the standard CSV field naming conventions.)

Parameters:

datasetname - - the name of the dataset to get dataset fields for. Can begin with '~' or not.

clusterName - - optional. If given, the name of the cluster associated with the dataset.

fieldSeparator - - optional. if given, and if the file being retrieved is a csv file, and if no field definitions exist for that csv,. this separator will be used rather than the separator defined during spraying when splitting up the first line of the file to define the number of fields

Returns:

an ArrayList of DFUDataColumns containing the name and field type.

Throws:

Exception

getFirstRow

public String getFirstRow(String datasetname,
                          String clustername)
                   throws Exception

Returns the first row of data for a dataset

Parameters:

datasetname - - logical filename, with or without '~' at the beginning

clustername - - optional. name of the cluster the file's associated with

Returns:

a String of data representing the first row in the file.

Throws:

Exception

getLogicalFiles

public List<DFULogicalFileInfo> getLogicalFiles(String filename,
                                                String cluster,
                                                int firstN,
                                                int pageStartFrom,
                                                int pageSize)
                                         throws Exception

Get array of logical files on target HPCC system based on input parameters

Parameters:

filename -

cluster - --- NO LONGER USED ---

firstN -

pageStartFrom -

pageSize -

Returns:

Throws:

Exception

getDFUData

public DFUSearchDataResponse getDFUData(String openLogicalName,
                                        String cluster,
                                        boolean roxieSelections,
                                        int chooseFile,
                                        int count,
                                        boolean schemaOnly,
                                        long startIndex)
                                 throws Exception

Return raw response from WsDFU DFUSearchData method requests

Parameters:

openLogicalName -

cluster -

roxieSelections -

chooseFile -

count -

schemaOnly -

startIndex -

Returns:

Throws:

Exception

isComplete

protected boolean isComplete()

Specified by:

isComplete in class DataSingleton

fastRefresh

protected void fastRefresh()

Specified by:

fastRefresh in class DataSingleton

fullRefresh

protected void fullRefresh()

Specified by:

fullRefresh in class DataSingleton

equals

public boolean equals(Object aThat)

Specified by:

equals in class DataSingleton

hashCode

public int hashCode()

Specified by:

hashCode in class DataSingleton

deleteFiles

public List<DFUResult> deleteFiles(Set<String> files,
                                   String cluster)
                            throws Exception

Parameters:

files - - list of filenames to delete

cluster - - name of cluster to delete from (will delete from all clusters if null)

Returns:

list of results of file deletions

Throws:

Exception

searchFiles

public List<DFULogicalFileInfo> searchFiles(String logicalFilename,
                                            String cluster)
                                     throws Exception

searchFiles

Parameters:

logicalFilename - - the filename to search for

cluster - - the cluster to search on

Returns:

- true if the file exists on the specified cluster (or on any cluster if the input cluster is null), false otherwise

Throws:

Exception

getFileAccessBlob

public String getFileAccessBlob(SecAccessType accesstype,
                                String filename,
                                String clustername,
                                int expiryseconds,
                                String jobid)
                         throws Exception

getFileAccessBlob - HPCC 7.0.x version The response is to be used in conjunction with DAFILESRV's rowservice distributed file read stream. The response grants the holder access to the target file for the duration of 'expiryseconds' seconds since the Accessblob is generated. This version is to be used when targeting a 7.0.x HPCC (ESP and DAFILESRV)

Parameters:

accesstype - - the file access level to request to request

filename - - the name of the target file to be accessed

clustername - - the name of the target file's HPCC cluster (can be empty)

expiryseconds - - the number of seconds file access is granted

jobid - - unique identifier for access token

Returns:

- Access artifact to be propagated as part of DAFILESERV file access requests

Throws:

Exception

getFileAccessBlob

public String getFileAccessBlob(String filename,
                                String clustername,
                                int expiryseconds,
                                String jobid)
                         throws Exception

getFileAccessBlob The response is to be used in conjunction with DAFILESRV's rowservice distributed file read/write stream. The response grants the holder access to the target file for the duration of 'expiryseconds' seconds since the Accessblob is generated.

Parameters:

filename - - the name of the target file to be accessed

clustername - - the name of the target file's HPCC cluster (can be empty)

expiryseconds - - the number of seconds file access is granted

jobid - - unique identifier for access token

Returns:

- Access artifact to be propagated as part of DAFILESERV file access requests

Throws:

Exception

getFileAccess

public DFUFileAccessInfoWrapper getFileAccess(SecAccessType accesstype,
                                              String filename,
                                              String clustername,
                                              int expiryseconds,
                                              String jobid,
                                              boolean includejsonTypeInfo,
                                              boolean includebinTypeInfo,
                                              boolean requestfileinfo)
                                       throws Exception

To be used in conjunction with DAFILESRV's rowservice distributed file read stream. The response wrapper provides access to the 'accessblob' which grants the holder read access to the target file for the duration of 'expiryseconds' seconds. Other access details can be fetched from the wrapper. since the Accessblob is generated.

Parameters:

accesstype - HPCC 7.0.x version - the file access level to request to request

filename - - the name of the target file to be accessed

clustername - - the name of the target file's HPCC cluster (can be empty)

expiryseconds - - the number of seconds file access is granted

jobid - - unique identifier for access token

includejsonTypeInfo - - flag to request file info in JSON format

includebinTypeInfo - - flag to request file info in Binary format

Returns:

- Access artifact to be propagated as part of DAFILESERV file access requests

Throws:

Exception

getFileAccess

public DFUFileAccessInfoWrapper getFileAccess(String filename,
                                              String clustername,
                                              int expiryseconds,
                                              String jobid)
                                       throws Exception

To be used in conjunction with DAFILESRV's rowservice distributed file read/write stream. The response wrapper provides access to the 'accessblob' which grants the holder read access to the target file for the duration of 'expiryseconds' seconds. Other access details can be fetched from the wrapper.

Parameters:

filename - - the name of the target file to be accessed

clustername - - the name of the target file's HPCC cluster (can be empty)

expiryseconds - - the number of seconds file access is granted

jobid - - unique identifier for access token

Returns:

- Access artifact to be propagated as part of DAFILESERV file access requests

Throws:

Exception

createFile

public DFUCreateFileWrapper createFile(String fileName,
                                       String cluster,
                                       String eclRecordDefinition,
                                       String[] partitionHostMap,
                                       int expirySeconds)
                                throws Exception

Create a new (unpublished) dfu file. Does not request all file metadata. DAFILESERV fileaccess token is requested

Parameters:

fileName -

cluster -

eclRecordDefinition -

partitionHostMap - Array declaring the Filepart[i] to Node mapping

expirySeconds -

Returns:

Throws:

Exception

createFileAndAcquireAccess

public DFUCreateFileWrapper createFileAndAcquireAccess(String fileName,
                                                       String cluster,
                                                       String eclRecordDefinition,
                                                       String[] partitionHostMap,
                                                       int expirySeconds,
                                                       Boolean returnBinTypeInfo,
                                                       Boolean returnJsonTypeInfo,
                                                       FileAccessRole accessRole,
                                                       SecAccessType accessType)
                                                throws Exception

Create a new (unpublished) dfu file. All file metadata can be requested. DAFILESERV fileaccess token is requested

Parameters:

fileName -

cluster -

eclRecordDefinition -

partitionHostMap - Array declaring the Filepart[i] to Node mapping

expirySeconds -

returnBinTypeInfo -

returnJsonTypeInfo -

accessRole -

accessType -

Returns:

Throws:

Exception

createFile

public DFUCreateFileWrapper createFile(String fileName,
                                       String cluster,
                                       String eclRecordDefinition,
                                       int expirySeconds)
                                throws Exception

Create a new (unpublished), uncompressed dfu file. JSON based info will be requested -- appropriate for HPCC post 7.2.0 DAFILESERV fileaccess token is requested

Parameters:

fileName -

cluster -

eclRecordDefinition -

expirySeconds -

Returns:

Throws:

Exception

createFile

public DFUCreateFileWrapper createFile(String fileName,
                                       String cluster,
                                       String eclRecordDefinition,
                                       int expirySeconds,
                                       Boolean compressed)
                                throws Exception

Create a new (unpublished) dfu file. JSON based info will be requested -- appropriate for HPCC post 7.2.0 DAFILESERV fileaccess token is requested

Parameters:

fileName -

cluster -

eclRecordDefinition -

expirySeconds -

compressed -

Returns:

Throws:

Exception

createFile

public DFUCreateFileWrapper createFile(String fileName,
                                       String cluster,
                                       String eclRecordDefinition,
                                       int expirySeconds,
                                       Boolean compressed,
                                       DFUFileTypeWrapper filetype,
                                       String requestid)
                                throws Exception

Create a new (unpublished) dfu file. JSON based info will be requested -- appropriate for HPCC post 7.2.0 DAFILESERV fileaccess token is requested

Parameters:

fileName -

cluster -

eclRecordDefinition -

expirySeconds -

compressed -

filetype - - for example DFUFileTypeWrapper.Csv

requestid -

Returns:

Throws:

Exception

createFileAndAcquireAccess

public DFUCreateFileWrapper createFileAndAcquireAccess(String fileName,
                                                       String cluster,
                                                       String eclRecordDefinition,
                                                       int expirySeconds,
                                                       Boolean compressed,
                                                       DFUFileTypeWrapper type,
                                                       String requestId)
                                                throws Exception

Create a new (unpublished) dfu file. JSON based info will be requested -- appropriate for HPCC post 7.2.0 DAFILESERV fileaccess token is requested

Parameters:

fileName -

cluster -

eclRecordDefinition -

expirySeconds -

compressed -

type - - for example DFUFileTypeWrapper.Csv

requestId -

Returns:

Throws:

Exception

publishFile

public void publishFile(String fileId,
                        String eclRecordDefinition,
                        long totalRecords,
                        long fileSize)
                 throws Exception

Publishes file created by WsDFUCreateFile. Requires fileId returned from createfile method call

Parameters:

fileId -

eclRecordDefinition -

totalRecords -

fileSize -

Throws:

Exception

publishFile

public void publishFile(String fileId,
                        String eclRecordDefinition,
                        long totalRecords,
                        long fileSize,
                        Boolean overwrite)
                 throws Exception

Publishes file created by WsDFUCreateFile. Requires fileId returned from createfile method call

Parameters:

fileId -

eclRecordDefinition -

totalRecords -

fileSize -

overwrite -

Throws:

Exception

getFileDetails

public DFUFileDetailInfo getFileDetails(String logicalname,
                                        String clustername)
                                 throws Exception

Parameters:

logicalname - - logical file to get file info for, can start with '~' or not

clustername - - Optional. If specified, the cluster on which to search for the file

Returns:

a DFUInfoResponse object containing the file info

Throws:

Exception

getFileDetails

public DFUFileDetailInfo getFileDetails(String logicalname,
                                        String clustername,
                                        boolean jsonTypeInfo,
                                        boolean binTypeInfo)
                                 throws Exception

Parameters:

logicalname - logical file for request, can start with ~

clustername - optional

jsonTypeInfo - true if record structure information in JSON format is to be returned

binTypeInfo - true if record structure information in binary format is to be returned

Returns:

DFUInfoResponse object containing the information

Throws:

Exception