Class FirestoreV1
- java.lang.Object
-
- org.apache.beam.sdk.io.gcp.firestore.FirestoreV1
-
@Immutable public final class FirestoreV1 extends java.lang.Object
FirestoreV1
provides an API which provides lifecycle managedPTransform
s for Cloud Firestore v1 API.This class is part of the Firestore Connector DSL and should be accessed via
FirestoreIO.v1()
.All
PTransform
s provided by this API useGcpOptions
onPipelineOptions
for credentials access and projectId resolution. As such, the lifecycle of gRPC clients and project information is scoped to the bundle level, not the worker level.Operations
Read
The currently supported read operations and their execution behavior are as follows:
RPC Execution Behavior Example Usage FirestoreV1.PartitionQuery
Parallel Streaming PCollection<
PartitionQueryRequest
> partitionQueryRequests = ...; PCollection<RunQueryRequest
> runQueryRequests = partitionQueryRequests .apply(FirestoreIO.v1().read().partitionQuery()
.build()); PCollection<RunQueryResponse
> runQueryResponses = runQueryRequests .apply(FirestoreIO.v1().read().runQuery()
.build());FirestoreV1.RunQuery
Sequential Streaming PCollection<
RunQueryRequest
> runQueryRequests = ...; PCollection<RunQueryResponse
> runQueryResponses = runQueryRequests .apply(FirestoreIO.v1().read().runQuery()
.build());FirestoreV1.BatchGetDocuments
Sequential Streaming PCollection<
BatchGetDocumentsRequest
> batchGetDocumentsRequests = ...; PCollection<BatchGetDocumentsResponse
> batchGetDocumentsResponses = batchGetDocumentsRequests .apply(FirestoreIO.v1().read().batchGetDocuments()
.build());FirestoreV1.ListCollectionIds
Sequential Paginated PCollection<
ListCollectionIdsRequest
> listCollectionIdsRequests = ...; PCollection<ListCollectionIdsResponse
> listCollectionIdsResponses = listCollectionIdsRequests .apply(FirestoreIO.v1().read().listCollectionIds()
.build());FirestoreV1.ListDocuments
Sequential Paginated PCollection<
ListDocumentsRequest
> listDocumentsRequests = ...; PCollection<ListDocumentsResponse
> listDocumentsResponses = listDocumentsRequests .apply(FirestoreIO.v1().read().listDocuments()
.build());PartitionQuery should be preferred over other options if at all possible, becuase it has the ability to parallelize execution of multiple queries for specific sub-ranges of the full results. When choosing the value to set for
PartitionQueryRequest.Builder.setPartitionCount(long)
, ensure you are picking a value this makes sense for your data set and your max number of workers. If you find that a partition query is taking a unexpectedly long time, try increasing the number of partitions. Depending on how large your dataset is increasing as much as 10x can significantly reduce total partition query wall time.You should only ever use ListDocuments if the use of
show_missing
is needed to access a document. RunQuery and PartitionQuery will always be faster if the use ofshow_missing
is not needed.Write
To write aPCollection
to Cloud Firestore usewrite()
, picking the behavior of the writer.Writes use Cloud Firestore's BatchWrite api which provides fine grained write semantics.
The default behavior is to fail a bundle if any single write fails with a non-retryable error.
PCollection<
Alternatively, if you'd rather output write failures to a Dead Letter Queue addWrite
> writes = ...; PCollection<FirestoreV1.WriteSuccessSummary
> sink = writes .apply(FirestoreIO.v1().write().batchWrite()
.build());withDeadLetterQueue
when building your writer.PCollection<
Write
> writes = ...; PCollection<FirestoreV1.WriteFailure
> writeFailures = writes .apply(FirestoreIO.v1().write().batchWrite()
.withDeadLetterQueue().build());Permissions
Permission requirements depend on thePipelineRunner
that is used to execute the pipeline. Please refer to the documentation of correspondingPipelineRunner
s for more details.Please see Security for server client libraries > Roles for security and permission related information specific to Cloud Firestore.
Optionally, Cloud Firestore V1 Emulator, running locally, could be used for testing purposes by providing the host port information via
FirestoreOptions.setEmulatorHost(String)
. In such a case, all the Cloud Firestore API calls are directed to the Emulator.- See Also:
FirestoreIO.v1()
,PipelineRunner
,PipelineOptions
,GcpOptions
, Cloud Firestore v1 API
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
FirestoreV1.BatchGetDocuments
Concrete class representing aPTransform
<
PCollection
<
BatchGetDocumentsRequest
>,
PTransform
<
BatchGetDocumentsResponse
>>
which will read from Firestore.static class
FirestoreV1.BatchWriteWithDeadLetterQueue
Concrete class representing aPTransform
<
PCollection
<
Write
>,
PCollection
<
FirestoreV1.WriteFailure
>
which will write to Firestore.static class
FirestoreV1.BatchWriteWithSummary
Concrete class representing aPTransform
<
PCollection
<
Write
>,
PDone
>
which will write to Firestore.static class
FirestoreV1.FailedWritesException
Exception that is thrown if one or moreWrite
s is unsuccessful with a non-retryable status code.static class
FirestoreV1.ListCollectionIds
Concrete class representing aPTransform
<
PCollection
<
ListCollectionIdsRequest
>,
PTransform
<
ListCollectionIdsResponse
>>
which will read from Firestore.static class
FirestoreV1.ListDocuments
Concrete class representing aPTransform
<
PCollection
<
ListDocumentsRequest
>,
PTransform
<
ListDocumentsResponse
>>
which will read from Firestore.static class
FirestoreV1.PartitionQuery
Concrete class representing aPTransform
<
PCollection
<
PartitionQueryRequest
>,
PTransform
<
RunQueryRequest
>>
which will read from Firestore.static class
FirestoreV1.Read
Type safe builder factory for read operations.static class
FirestoreV1.RunQuery
Concrete class representing aPTransform
<
PCollection
<
RunQueryRequest
>,
PTransform
<
RunQueryResponse
>>
which will read from Firestore.static class
FirestoreV1.Write
Type safe builder factory for write operations.static class
FirestoreV1.WriteFailure
Failure details for an attemptedWrite
.static class
FirestoreV1.WriteSuccessSummary
Summary object produced when a number of writes are successfully written to Firestore in a single BatchWrite.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description FirestoreV1.Read
read()
The class returned by this method provides the ability to createPTransforms
for read operations available in the Firestore V1 API provided byFirestoreStub
.FirestoreV1.Write
write()
The class returned by this method provides the ability to createPTransforms
for write operations available in the Firestore V1 API provided byFirestoreStub
.
-
-
-
Method Detail
-
read
public FirestoreV1.Read read()
The class returned by this method provides the ability to createPTransforms
for read operations available in the Firestore V1 API provided byFirestoreStub
.This method is part of the Firestore Connector DSL and should be accessed via
FirestoreIO.v1()
.- Returns:
- Type safe builder factory for read operations.
- See Also:
FirestoreIO.v1()
-
write
public FirestoreV1.Write write()
The class returned by this method provides the ability to createPTransforms
for write operations available in the Firestore V1 API provided byFirestoreStub
.This method is part of the Firestore Connector DSL and should be accessed via
FirestoreIO.v1()
.- Returns:
- Type safe builder factory for write operations.
- See Also:
FirestoreIO.v1()
-
-