Package com.cognite.client
Class Files
- java.lang.Object
-
- com.cognite.client.Files
-
-
Field Summary
Fields Modifier and Type Field Description protected static org.slf4j.Logger
LOG
-
Constructor Summary
Constructors Constructor Description Files()
-
Method Summary
All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description protected Request
addAuthInfo(Request request)
Adds the required authentication information into the request object.protected Aggregate
aggregate(ResourceType resourceType, Request requestParameters)
Performs an item aggregation request to Cognite Data Fusion.Aggregate
aggregate(Request requestParameters)
Performs an item aggregation request to Cognite Data Fusion.protected List<String>
buildPartitionsList(int noPartitions)
Builds an array of partition specifications for parallel retrieval from the Cognite api.protected List<Item>
deDuplicate(Collection<Item> itemList)
De-duplicates a collection ofItem
.List<Item>
delete(List<Item> files)
Deletes a set of files.List<FileContainer>
download(List<Item> files, Path downloadPath, boolean preferByteStream)
Downloads file binaries.List<FileBinary>
downloadFileBinaries(List<Item> fileItems, URI tempStoragePath, boolean forceTempStorage)
Downloads file binaries.List<FileContainer>
downloadToPath(List<Item> files, Path downloadPath)
Downloads file binaries to a local / network path.abstract CogniteClient
getClient()
protected Iterator<CompletableFuture<ResponseItems<String>>>
getListResponseIterator(ResourceType resourceType, Request requestParameters)
protected boolean
itemsHaveId(Collection<Item> items)
Returns true if all items contain either an externalId or id.Iterator<List<FileMetadata>>
list(Request requestParameters)
Returns allFileMetadata
objects that matches the filters set in theRequest
.Iterator<List<FileMetadata>>
list(Request requestParameters, String... partitions)
protected Iterator<List<String>>
listJson(ResourceType resourceType, Request requestParameters, String... partitions)
Will return the results from alist / filter
api endpoint.protected Iterator<List<String>>
listJson(ResourceType resourceType, Request requestParameters, String partitionKey, String... partitions)
Will return the results from alist / filter
api endpoint.protected Map<String,Item>
mapItemToId(Collection<Item> items)
Maps all items to their externalId (primary) or id (secondary).static Files
of(CogniteClient client)
Constructs a newFiles
object using the provided client configuration.protected List<Item>
parseItems(List<String> input)
Parses a list of item object in json representation to typed objects.protected String
parseName(String json)
Returns the name attribute value from a json input.protected String
parseString(String itemJson, String fieldName)
Try parsing the specified Json path as aString
.List<FileMetadata>
retrieve(List<Item> items)
Retrieve files by id.protected List<String>
retrieveJson(ResourceType resourceType, Collection<Item> items)
Retrieve items by id.protected List<Map<String,Object>>
toRequestItems(Collection<Item> itemList)
Converts a list ofItem
to a request object structure (that can later be parsed to Json).List<FileMetadata>
upload(List<FileContainer> files)
Uploads a set of file headers and binaries to Cognite Data Fusion.List<FileMetadata>
upload(List<FileContainer> files, boolean deleteTempFile)
Uploads a set of file headers and binaries to Cognite Data Fusion.List<FileMetadata>
upsert(List<FileMetadata> fileMetadataList)
Creates or updates a set ofFileMetadata
objects.
-
-
-
Method Detail
-
of
public static Files of(CogniteClient client)
Constructs a newFiles
object using the provided client configuration. This method is intended for internal use--SDK clients should always useCogniteClient
as the entry point to this class.- Parameters:
client
- TheCogniteClient
to use for configuration settings.- Returns:
- the assets api object.
-
list
public Iterator<List<FileMetadata>> list(Request requestParameters) throws Exception
Returns allFileMetadata
objects that matches the filters set in theRequest
. The results are paged through / iterated over via anIterator
--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure. The assets are retrieved using multiple, parallel request streams towards the Cognite api. The number of parallel streams are set in theClientConfig
.
-
list
public Iterator<List<FileMetadata>> list(Request requestParameters, String... partitions) throws Exception
Returns allEvent
objects that matches the filters set in theRequest
for the specified partitions. This is method is intended for advanced use cases where you need direct control over the individual partitions. For example, when using the SDK in a distributed computing environment. The results are paged through / iterated over via anIterator
--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure.
-
retrieve
public List<FileMetadata> retrieve(List<Item> items) throws Exception
Retrieve files by id.- Parameters:
items
- The item(s)externalId / id
to retrieve.- Returns:
- The retrieved file headers.
- Throws:
Exception
-
aggregate
public Aggregate aggregate(Request requestParameters) throws Exception
Performs an item aggregation request to Cognite Data Fusion. The default aggregation is a total item count based on the (optional) filters in the request. Multiple aggregation types are supported. Please refer to the Cognite API specification for more information on the possible settings.- Parameters:
requestParameters
- The filtering and aggregates specification- Returns:
- The aggregation results.
- Throws:
Exception
- See Also:
- Cognite API v1 specification
-
upsert
public List<FileMetadata> upsert(List<FileMetadata> fileMetadataList) throws Exception
Creates or updates a set ofFileMetadata
objects. If it is a newFileMetadata
object (based onid / externalId
, then it will be created. If anFileMetadata
object already exists in Cognite Data Fusion, it will be updated. The update behavior is specified via the update mode in theClientConfig
settings.- Parameters:
fileMetadataList
- The file headers / metadata to upsert.- Returns:
- The upserted file headers.
- Throws:
Exception
-
upload
public List<FileMetadata> upload(List<FileContainer> files) throws Exception
Uploads a set of file headers and binaries to Cognite Data Fusion. The file binary can either be placed in-memory in the file container (as a byte string) or referenced (by URI) to a blob store.- Parameters:
files
- The files to upload.- Returns:
- The file metadata / headers for the uploaded files.
- Throws:
Exception
-
upload
public List<FileMetadata> upload(@NotNull List<FileContainer> files, boolean deleteTempFile) throws Exception
Uploads a set of file headers and binaries to Cognite Data Fusion. The file binary can either be placed in-memory in the file container (as a byte string) or referenced (by URI) to a blob store. In case you reference the file by URI, you can choose to automatically remove the file binary from the (URI referenced) blob store after a successful upload to Cognite Data Fusion. This can Be useful in situations where you perform large scala data transfers utilizing a temp backing store.- Parameters:
files
- The files to upload.deleteTempFile
- Set to true to remove the URI binary after upload. Set to false to keep the URI binary.- Returns:
- The file metadata / headers for the uploaded files.
- Throws:
Exception
-
download
public List<FileContainer> download(List<Item> files, Path downloadPath, boolean preferByteStream) throws Exception
Downloads file binaries. Downloads a set of file binaries based onexternalId / id
in theItem
list. The file binaries can be downloaded as files or byte streams. In case the file is very large (> 200MB) it has to be streamed directly to the file system (i.e. downloaded as a file). Both the file header / metadata and the file binary will be returned. The complete information is encapsulated int theFileContainer
returned from this method. TheFileContainer
will host the file binary stream if you setpreferByteStream
totrue
and the file size is < 200 MB. IfpreferByteStream
is set tofalse
or the file size is > 200MB the file binary will be stored on disk and theFileContainer
will return theURI
reference to the binary. Supported destination file stores for the file binary: - Local (network) disk. Specify the temp path asfile://<host>/<my-path>/
. Examples:file://localhost/home/files/, file:///home/files/, file:///c:/temp/
- Google Cloud Storage. Specify the temp path asgs://<my-storage-bucket>/<my-path>/
.- Parameters:
files
- The list of files to download.downloadPath
- The URI to the download storagepreferByteStream
- Set to true to return byte streams when possible, set to false to always store binary as file.- Returns:
- File containers with file headers and references/byte streams of the binary.
- Throws:
Exception
-
downloadToPath
public List<FileContainer> downloadToPath(List<Item> files, Path downloadPath) throws Exception
Downloads file binaries to a local / network path. Downloads a set of file binaries based onexternalId / id
in theItem
list. Both the file header / metadata and the file binary will be returned. The complete information is encapsulated int theFileContainer
returned from this method. TheFileContainer
will host theURI
reference to the binary. Supported destination file stores for the file binary: - Local (network) disk. Specify the temp path asfile://<host>/<my-path>/
. Examples:file://localhost/home/files/, file:///home/files/, file:///c:/temp/
- Google Cloud Storage. Specify the temp path asgs://<my-storage-bucket>/<my-path>/
.- Parameters:
files
- The list of files to download.downloadPath
- The URI to the download storage- Returns:
- File containers with file headers and references/byte streams of the binary.
- Throws:
Exception
-
downloadFileBinaries
public List<FileBinary> downloadFileBinaries(List<Item> fileItems, @Nullable URI tempStoragePath, boolean forceTempStorage) throws Exception
Downloads file binaries. This method is intended for advanced use cases, for example when using this SDK as a part of a distributed system. Downloads a set of file binaries based onexternalId / id
in theItem
list. The file binaries can be downloaded as files or byte streams. In case the file is very large (> 200MB) it has to be streamed directly to the file system (to the temp storage area). Supported temp storage for the file binary: - Local (network) disk. Specify the temp path asfile://<host>/<my-path>/
. Examples:file://localhost/home/files/, file:///home/files/, file:///c:/temp/
- Google Cloud Storage. Specify the temp path asgs://<my-storage-bucket>/<my-path>/
.- Parameters:
fileItems
- The list of files to download.tempStoragePath
- The URI to the download storage. Set to null to only perform in-memory download.forceTempStorage
- Set to true to always download the binary to temp storage- Returns:
- The file binary.
- Throws:
Exception
-
delete
public List<Item> delete(List<Item> files) throws Exception
Deletes a set of files. The files to delete are identified via theirexternalId / id
by submitting a list ofItem
.
-
getClient
public abstract CogniteClient getClient()
-
buildPartitionsList
protected List<String> buildPartitionsList(int noPartitions)
Builds an array of partition specifications for parallel retrieval from the Cognite api. This specification is used as a parameter together with the filter / list endpoints. The number of partitions indicate the number of parallel read streams. Employ one partition specification per read stream.- Parameters:
noPartitions
- The total number of partitions- Returns:
- a
List
of partition specifications
-
listJson
protected Iterator<List<String>> listJson(ResourceType resourceType, Request requestParameters, String... partitions) throws Exception
Will return the results from alist / filter
api endpoint. For example, thefilter assets
endpoint. The results are paged through / iterated over via anIterator
--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure. This method support parallel retrieval via a set ofpartition
specifications. The specified partitions will be collected and merged together before being returned via theIterator
.- Parameters:
resourceType
- The resource type to query / filter / list. Ex.event, asset, time series
.requestParameters
- The query / filter specification. Follows the Cognite api request parameters.partitions
- An optional set of partitions to read via.- Returns:
- an
Iterator
over the results set. - Throws:
Exception
-
listJson
protected Iterator<List<String>> listJson(ResourceType resourceType, Request requestParameters, String partitionKey, String... partitions) throws Exception
Will return the results from alist / filter
api endpoint. For example, thefilter assets
endpoint. The results are paged through / iterated over via anIterator
--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure. This method support parallel retrieval via a set ofpartition
specifications. The specified partitions will be collected and merged together before being returned via theIterator
.- Parameters:
resourceType
- The resource type to query / filter / list. Ex.event, asset, time series
.requestParameters
- The query / filter specification. Follows the Cognite api request parameters.partitionKey
- The key to use for the partitions in the read request. For examplepartition
orcursor
.partitions
- An optional set of partitions to read via.- Returns:
- an
Iterator
over the results set. - Throws:
Exception
-
retrieveJson
protected List<String> retrieveJson(ResourceType resourceType, Collection<Item> items) throws Exception
Retrieve items by id.
-
aggregate
protected Aggregate aggregate(ResourceType resourceType, Request requestParameters) throws Exception
Performs an item aggregation request to Cognite Data Fusion. The default aggregation is a total item count based on the (optional) filters in the request. Some resource types, for exampleEvent
, supports multiple types of aggregation.- Parameters:
resourceType
- The resource type to perform aggregation of.requestParameters
- The request containing filters.- Returns:
- The aggregation result.
- Throws:
Exception
- See Also:
- Cognite API v1 specification
-
addAuthInfo
protected Request addAuthInfo(Request request) throws Exception
Adds the required authentication information into the request object. If the request object already have complete auth info nothing will be added. The following authentication schemes are supported: 1) API key. When using an api key, this service will look up the corresponding project/tenant to issue requests to.- Parameters:
request
- The request to enrich with auth information.- Returns:
- The request parameters with auth info added to it.
- Throws:
Exception
-
getListResponseIterator
protected Iterator<CompletableFuture<ResponseItems<String>>> getListResponseIterator(ResourceType resourceType, Request requestParameters) throws Exception
- Throws:
Exception
-
parseItems
protected List<Item> parseItems(List<String> input) throws Exception
Parses a list of item object in json representation to typed objects.- Parameters:
input
- the item list in Json string representation- Returns:
- the parsed item objects
- Throws:
Exception
-
toRequestItems
protected List<Map<String,Object>> toRequestItems(Collection<Item> itemList)
Converts a list ofItem
to a request object structure (that can later be parsed to Json).- Parameters:
itemList
- The items to parse.- Returns:
- The items in request item object form.
-
deDuplicate
protected List<Item> deDuplicate(Collection<Item> itemList)
De-duplicates a collection ofItem
.- Parameters:
itemList
-- Returns:
-
itemsHaveId
protected boolean itemsHaveId(Collection<Item> items)
Returns true if all items contain either an externalId or id.- Parameters:
items
-- Returns:
-
mapItemToId
protected Map<String,Item> mapItemToId(Collection<Item> items)
Maps all items to their externalId (primary) or id (secondary). If the id function does not return any identity, the item will be mapped to the empty string. Via the identity mapping, this function will also perform deduplication of the input items.- Parameters:
items
- the items to map to externalId / id.- Returns:
- the
Map
with all items mapped to externalId / id.
-
parseString
protected String parseString(String itemJson, String fieldName)
Try parsing the specified Json path as aString
.- Parameters:
itemJson
- The Json stringfieldName
- The Json path to parse- Returns:
- The Json path as a
String
.
-
-