Class FakeDatasetService
- java.lang.Object
-
- org.apache.beam.sdk.io.gcp.testing.FakeDatasetService
-
- All Implemented Interfaces:
java.io.Serializable
,java.lang.AutoCloseable
,BigQueryServices.DatasetService
@Internal public class FakeDatasetService extends java.lang.Object implements BigQueryServices.DatasetService, java.io.Serializable
A fake dataset service that can be serialized, for use in testReadFromTable.- See Also:
- Serialized Form
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from interface org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.DatasetService
BigQueryServices.DatasetService.TableMetadataView
-
-
Constructor Summary
Constructors Constructor Description FakeDatasetService()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse>
commitWriteStreams(java.lang.String tableUrn, java.lang.Iterable<java.lang.String> writeStreamNames)
Commit write streams of type PENDING.void
createDataset(java.lang.String projectId, java.lang.String datasetId, java.lang.String location, java.lang.String description, java.lang.Long defaultTableExpirationMs)
Create aDataset
with the givenlocation
,description
and default expiration time for tables in the dataset (ifnull
, tables don't expire).void
createTable(com.google.api.services.bigquery.model.Table table)
Creates the specified table if it does not exist.com.google.cloud.bigquery.storage.v1.WriteStream
createWriteStream(java.lang.String tableUrn, com.google.cloud.bigquery.storage.v1.WriteStream.Type type)
Create a Write Stream for use with the Storage Write API.void
deleteDataset(java.lang.String projectId, java.lang.String datasetId)
Deletes the dataset specified by the datasetId value.void
deleteTable(com.google.api.services.bigquery.model.TableReference tableRef)
Deletes the table specified by tableId from the dataset.void
failOnInsert(java.util.Map<com.google.api.services.bigquery.model.TableRow,java.util.List<com.google.api.services.bigquery.model.TableDataInsertAllResponse.InsertErrors>> insertErrors)
Cause a givenTableRow
object to fail when it's inserted.com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FinalizeWriteStreamResponse>
finalizeWriteStream(java.lang.String streamName)
Finalize a write stream.com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FlushRowsResponse>
flush(java.lang.String streamName, long offset)
Flush a given stream up to the given offset.java.util.List<java.lang.String>
getAllIds(java.lang.String projectId, java.lang.String datasetId, java.lang.String tableId)
java.util.List<com.google.api.services.bigquery.model.TableRow>
getAllRows(java.lang.String projectId, java.lang.String datasetId, java.lang.String tableId)
com.google.api.services.bigquery.model.Dataset
getDataset(java.lang.String projectId, java.lang.String datasetId)
Gets the specifiedDataset
resource by dataset ID.int
getInsertCount()
BigQueryServices.StreamAppendClient
getStreamAppendClient(java.lang.String streamName, com.google.protobuf.Descriptors.Descriptor descriptor, boolean useConnectionPool)
Create an append client for a given Storage API write stream.com.google.api.services.bigquery.model.Table
getTable(com.google.api.services.bigquery.model.TableReference tableRef)
Gets the specifiedTable
resource by table ID.com.google.api.services.bigquery.model.Table
getTable(com.google.api.services.bigquery.model.TableReference tableRef, java.util.List<java.lang.String> selectedFields)
com.google.api.services.bigquery.model.Table
getTable(com.google.api.services.bigquery.model.TableReference tableRef, java.util.List<java.lang.String> selectedFields, BigQueryServices.DatasetService.TableMetadataView view)
com.google.api.services.bigquery.model.Table
getTableImpl(com.google.api.services.bigquery.model.TableReference tableRef, java.util.List<java.lang.String> selectedFields, BigQueryServices.DatasetService.TableMetadataView view)
long
insertAll(com.google.api.services.bigquery.model.TableReference ref, java.util.List<com.google.api.services.bigquery.model.TableRow> rowList, java.util.List<java.lang.String> insertIdList)
<T> long
insertAll(com.google.api.services.bigquery.model.TableReference ref, java.util.List<org.apache.beam.sdk.values.FailsafeValueInSingleWindow<com.google.api.services.bigquery.model.TableRow,com.google.api.services.bigquery.model.TableRow>> rowList, java.util.List<java.lang.String> insertIdList, InsertRetryPolicy retryPolicy, java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<T>> failedInserts, ErrorContainer<T> errorContainer, boolean skipInvalidRows, boolean ignoreUnknownValues, boolean ignoreInsertIds, java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<com.google.api.services.bigquery.model.TableRow>> successfulRows)
InsertsTableRows
with the specified insertIds if not null.boolean
isTableEmpty(com.google.api.services.bigquery.model.TableReference tableRef)
Returns true if the table is empty.com.google.api.services.bigquery.model.Table
patchTableDescription(com.google.api.services.bigquery.model.TableReference tableReference, java.lang.String tableDescription)
Patch BigQueryTable
description.void
setShouldFailRow(java.util.function.Function<com.google.api.services.bigquery.model.TableRow,java.lang.Boolean> shouldFailRow)
static void
setUp()
void
updateTableSchema(com.google.api.services.bigquery.model.TableReference tableReference, com.google.api.services.bigquery.model.TableSchema tableSchema)
-
-
-
Method Detail
-
close
public void close() throws java.lang.Exception
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Throws:
java.lang.Exception
-
setUp
public static void setUp()
-
setShouldFailRow
public void setShouldFailRow(java.util.function.Function<com.google.api.services.bigquery.model.TableRow,java.lang.Boolean> shouldFailRow)
-
getTable
public com.google.api.services.bigquery.model.Table getTable(com.google.api.services.bigquery.model.TableReference tableRef) throws java.lang.InterruptedException, java.io.IOException
Description copied from interface:BigQueryServices.DatasetService
Gets the specifiedTable
resource by table ID.Returns null if the table is not found.
- Specified by:
getTable
in interfaceBigQueryServices.DatasetService
- Throws:
java.lang.InterruptedException
java.io.IOException
-
getTable
public com.google.api.services.bigquery.model.Table getTable(com.google.api.services.bigquery.model.TableReference tableRef, @Nullable java.util.List<java.lang.String> selectedFields) throws java.lang.InterruptedException, java.io.IOException
- Specified by:
getTable
in interfaceBigQueryServices.DatasetService
- Throws:
java.lang.InterruptedException
java.io.IOException
-
getTable
public com.google.api.services.bigquery.model.Table getTable(com.google.api.services.bigquery.model.TableReference tableRef, @Nullable java.util.List<java.lang.String> selectedFields, @Nullable BigQueryServices.DatasetService.TableMetadataView view) throws java.lang.InterruptedException, java.io.IOException
- Specified by:
getTable
in interfaceBigQueryServices.DatasetService
- Throws:
java.lang.InterruptedException
java.io.IOException
-
getTableImpl
public com.google.api.services.bigquery.model.Table getTableImpl(com.google.api.services.bigquery.model.TableReference tableRef, @Nullable java.util.List<java.lang.String> selectedFields, @Nullable BigQueryServices.DatasetService.TableMetadataView view) throws java.lang.InterruptedException, java.io.IOException
- Throws:
java.lang.InterruptedException
java.io.IOException
-
getAllRows
public java.util.List<com.google.api.services.bigquery.model.TableRow> getAllRows(java.lang.String projectId, java.lang.String datasetId, java.lang.String tableId) throws java.lang.InterruptedException, java.io.IOException
- Throws:
java.lang.InterruptedException
java.io.IOException
-
getAllIds
public java.util.List<java.lang.String> getAllIds(java.lang.String projectId, java.lang.String datasetId, java.lang.String tableId) throws java.lang.InterruptedException, java.io.IOException
- Throws:
java.lang.InterruptedException
java.io.IOException
-
deleteTable
public void deleteTable(com.google.api.services.bigquery.model.TableReference tableRef) throws java.io.IOException, java.lang.InterruptedException
Description copied from interface:BigQueryServices.DatasetService
Deletes the table specified by tableId from the dataset. If the table contains data, all the data will be deleted.- Specified by:
deleteTable
in interfaceBigQueryServices.DatasetService
- Throws:
java.io.IOException
java.lang.InterruptedException
-
createTable
public void createTable(com.google.api.services.bigquery.model.Table table) throws java.io.IOException
Description copied from interface:BigQueryServices.DatasetService
Creates the specified table if it does not exist.- Specified by:
createTable
in interfaceBigQueryServices.DatasetService
- Throws:
java.io.IOException
-
updateTableSchema
public void updateTableSchema(com.google.api.services.bigquery.model.TableReference tableReference, com.google.api.services.bigquery.model.TableSchema tableSchema) throws java.io.IOException
- Throws:
java.io.IOException
-
isTableEmpty
public boolean isTableEmpty(com.google.api.services.bigquery.model.TableReference tableRef) throws java.io.IOException, java.lang.InterruptedException
Description copied from interface:BigQueryServices.DatasetService
Returns true if the table is empty.- Specified by:
isTableEmpty
in interfaceBigQueryServices.DatasetService
- Throws:
java.io.IOException
- if the table is not found.java.lang.InterruptedException
-
getDataset
public com.google.api.services.bigquery.model.Dataset getDataset(java.lang.String projectId, java.lang.String datasetId) throws java.io.IOException, java.lang.InterruptedException
Description copied from interface:BigQueryServices.DatasetService
Gets the specifiedDataset
resource by dataset ID.- Specified by:
getDataset
in interfaceBigQueryServices.DatasetService
- Throws:
java.io.IOException
java.lang.InterruptedException
-
createDataset
public void createDataset(java.lang.String projectId, java.lang.String datasetId, java.lang.String location, java.lang.String description, java.lang.Long defaultTableExpirationMs) throws java.io.IOException, java.lang.InterruptedException
Description copied from interface:BigQueryServices.DatasetService
Create aDataset
with the givenlocation
,description
and default expiration time for tables in the dataset (ifnull
, tables don't expire).- Specified by:
createDataset
in interfaceBigQueryServices.DatasetService
- Throws:
java.io.IOException
java.lang.InterruptedException
-
deleteDataset
public void deleteDataset(java.lang.String projectId, java.lang.String datasetId) throws java.io.IOException, java.lang.InterruptedException
Description copied from interface:BigQueryServices.DatasetService
Deletes the dataset specified by the datasetId value.Before you can delete a dataset, you must delete all its tables.
- Specified by:
deleteDataset
in interfaceBigQueryServices.DatasetService
- Throws:
java.io.IOException
java.lang.InterruptedException
-
getInsertCount
public int getInsertCount()
-
insertAll
public long insertAll(com.google.api.services.bigquery.model.TableReference ref, java.util.List<com.google.api.services.bigquery.model.TableRow> rowList, @Nullable java.util.List<java.lang.String> insertIdList) throws java.io.IOException, java.lang.InterruptedException
- Throws:
java.io.IOException
java.lang.InterruptedException
-
insertAll
public <T> long insertAll(com.google.api.services.bigquery.model.TableReference ref, java.util.List<org.apache.beam.sdk.values.FailsafeValueInSingleWindow<com.google.api.services.bigquery.model.TableRow,com.google.api.services.bigquery.model.TableRow>> rowList, @Nullable java.util.List<java.lang.String> insertIdList, InsertRetryPolicy retryPolicy, java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<T>> failedInserts, ErrorContainer<T> errorContainer, boolean skipInvalidRows, boolean ignoreUnknownValues, boolean ignoreInsertIds, java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<com.google.api.services.bigquery.model.TableRow>> successfulRows) throws java.io.IOException, java.lang.InterruptedException
Description copied from interface:BigQueryServices.DatasetService
InsertsTableRows
with the specified insertIds if not null.If any insert fail permanently according to the retry policy, those rows are added to failedInserts.
Returns the total bytes count of
TableRows
.- Specified by:
insertAll
in interfaceBigQueryServices.DatasetService
- Throws:
java.io.IOException
java.lang.InterruptedException
-
patchTableDescription
public com.google.api.services.bigquery.model.Table patchTableDescription(com.google.api.services.bigquery.model.TableReference tableReference, @Nullable java.lang.String tableDescription) throws java.io.IOException, java.lang.InterruptedException
Description copied from interface:BigQueryServices.DatasetService
Patch BigQueryTable
description.- Specified by:
patchTableDescription
in interfaceBigQueryServices.DatasetService
- Throws:
java.io.IOException
java.lang.InterruptedException
-
createWriteStream
public com.google.cloud.bigquery.storage.v1.WriteStream createWriteStream(java.lang.String tableUrn, com.google.cloud.bigquery.storage.v1.WriteStream.Type type) throws java.io.IOException, java.lang.InterruptedException
Description copied from interface:BigQueryServices.DatasetService
Create a Write Stream for use with the Storage Write API.- Specified by:
createWriteStream
in interfaceBigQueryServices.DatasetService
- Throws:
java.io.IOException
java.lang.InterruptedException
-
getStreamAppendClient
public BigQueryServices.StreamAppendClient getStreamAppendClient(java.lang.String streamName, com.google.protobuf.Descriptors.Descriptor descriptor, boolean useConnectionPool)
Description copied from interface:BigQueryServices.DatasetService
Create an append client for a given Storage API write stream. The stream must be created first.- Specified by:
getStreamAppendClient
in interfaceBigQueryServices.DatasetService
-
flush
public com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FlushRowsResponse> flush(java.lang.String streamName, long offset)
Description copied from interface:BigQueryServices.DatasetService
Flush a given stream up to the given offset. The stream must have type BUFFERED.- Specified by:
flush
in interfaceBigQueryServices.DatasetService
-
finalizeWriteStream
public com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FinalizeWriteStreamResponse> finalizeWriteStream(java.lang.String streamName)
Description copied from interface:BigQueryServices.DatasetService
Finalize a write stream. After finalization, no more records can be appended to the stream.- Specified by:
finalizeWriteStream
in interfaceBigQueryServices.DatasetService
-
commitWriteStreams
public com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse> commitWriteStreams(java.lang.String tableUrn, java.lang.Iterable<java.lang.String> writeStreamNames)
Description copied from interface:BigQueryServices.DatasetService
Commit write streams of type PENDING. The streams must be finalized before committing.- Specified by:
commitWriteStreams
in interfaceBigQueryServices.DatasetService
-
failOnInsert
public void failOnInsert(java.util.Map<com.google.api.services.bigquery.model.TableRow,java.util.List<com.google.api.services.bigquery.model.TableDataInsertAllResponse.InsertErrors>> insertErrors)
Cause a givenTableRow
object to fail when it's inserted. The errors link the list will be returned on subsequent retries, and the insert will succeed when the errors run out.
-
-