software.amazon.awssdk.services.glue.model

Interface S3ParquetSource.Builder

All Superinterfaces:

Buildable, CopyableBuilder<S3ParquetSource.Builder,S3ParquetSource>, SdkBuilder<S3ParquetSource.Builder,S3ParquetSource>, SdkPojo

Enclosing class:

S3ParquetSource

public static interface S3ParquetSource.Builder
extends SdkPojo, CopyableBuilder<S3ParquetSource.Builder,S3ParquetSource>

Method Summary

Modifier and Type	Method and Description
default S3ParquetSource.Builder	additionalOptions(Consumer<S3DirectSourceAdditionalOptions.Builder> additionalOptions) Specifies additional connection options.
S3ParquetSource.Builder	additionalOptions(S3DirectSourceAdditionalOptions additionalOptions) Specifies additional connection options.
S3ParquetSource.Builder	compressionType(ParquetCompressionType compressionType) Specifies how the data is compressed.
S3ParquetSource.Builder	compressionType(String compressionType) Specifies how the data is compressed.
S3ParquetSource.Builder	exclusions(Collection<String> exclusions) A string containing a JSON list of Unix-style glob patterns to exclude.
S3ParquetSource.Builder	exclusions(String... exclusions) A string containing a JSON list of Unix-style glob patterns to exclude.
S3ParquetSource.Builder	groupFiles(String groupFiles) Grouping files is turned on by default when the input contains more than 50,000 files.
S3ParquetSource.Builder	groupSize(String groupSize) The target group size in bytes.
S3ParquetSource.Builder	maxBand(Integer maxBand) This option controls the duration in milliseconds after which the s3 listing is likely to be consistent.
S3ParquetSource.Builder	maxFilesInBand(Integer maxFilesInBand) This option specifies the maximum number of files to save from the last maxBand seconds.
S3ParquetSource.Builder	name(String name) The name of the data store.
S3ParquetSource.Builder	outputSchemas(Collection<GlueSchema> outputSchemas) Specifies the data schema for the S3 Parquet source.
S3ParquetSource.Builder	outputSchemas(Consumer<GlueSchema.Builder>... outputSchemas) Specifies the data schema for the S3 Parquet source.
S3ParquetSource.Builder	outputSchemas(GlueSchema... outputSchemas) Specifies the data schema for the S3 Parquet source.
S3ParquetSource.Builder	paths(Collection<String> paths) A list of the Amazon S3 paths to read from.
S3ParquetSource.Builder	paths(String... paths) A list of the Amazon S3 paths to read from.
S3ParquetSource.Builder	recurse(Boolean recurse) If set to true, recursively reads files in all subdirectories under the specified paths.

Methods inherited from interface software.amazon.awssdk.core.SdkPojo

equalsBySdkFields, sdkFields

Methods inherited from interface software.amazon.awssdk.utils.builder.CopyableBuilder

copy

Methods inherited from interface software.amazon.awssdk.utils.builder.SdkBuilder

applyMutation, build

Method Detail

name

S3ParquetSource.Builder name(String name)

The name of the data store.

Parameters:

name - The name of the data store.

Returns:

Returns a reference to this object so that method calls can be chained together.

paths

S3ParquetSource.Builder paths(Collection<String> paths)

A list of the Amazon S3 paths to read from.

Parameters:

paths - A list of the Amazon S3 paths to read from.

Returns:

Returns a reference to this object so that method calls can be chained together.

paths

S3ParquetSource.Builder paths(String... paths)

A list of the Amazon S3 paths to read from.

Parameters:

paths - A list of the Amazon S3 paths to read from.

Returns:

Returns a reference to this object so that method calls can be chained together.

compressionType

S3ParquetSource.Builder compressionType(String compressionType)

Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

Parameters:

compressionType - Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

Returns:

Returns a reference to this object so that method calls can be chained together.

See Also:

ParquetCompressionType, ParquetCompressionType

compressionType

S3ParquetSource.Builder compressionType(ParquetCompressionType compressionType)

Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

Parameters:

compressionType - Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

Returns:

Returns a reference to this object so that method calls can be chained together.

See Also:

ParquetCompressionType, ParquetCompressionType

exclusions

S3ParquetSource.Builder exclusions(Collection<String> exclusions)

A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.

Parameters:

exclusions - A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.

Returns:

Returns a reference to this object so that method calls can be chained together.

exclusions

S3ParquetSource.Builder exclusions(String... exclusions)

A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.

Parameters:

exclusions - A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.

Returns:

Returns a reference to this object so that method calls can be chained together.

groupSize

S3ParquetSource.Builder groupSize(String groupSize)

The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles" must be set to "inPartition" for this to take effect.

Parameters:

groupSize - The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles" must be set to "inPartition" for this to take effect.

Returns:

Returns a reference to this object so that method calls can be chained together.

groupFiles

S3ParquetSource.Builder groupFiles(String groupFiles)

Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none".

Parameters:

groupFiles - Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none".

Returns:

Returns a reference to this object so that method calls can be chained together.

recurse

S3ParquetSource.Builder recurse(Boolean recurse)

If set to true, recursively reads files in all subdirectories under the specified paths.

Parameters:

recurse - If set to true, recursively reads files in all subdirectories under the specified paths.

Returns:

Returns a reference to this object so that method calls can be chained together.

maxBand

S3ParquetSource.Builder maxBand(Integer maxBand)

This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.

Parameters:

maxBand - This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.

Returns:

Returns a reference to this object so that method calls can be chained together.

maxFilesInBand

S3ParquetSource.Builder maxFilesInBand(Integer maxFilesInBand)

This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.

Parameters:

maxFilesInBand - This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.

Returns:

Returns a reference to this object so that method calls can be chained together.

additionalOptions

S3ParquetSource.Builder additionalOptions(S3DirectSourceAdditionalOptions additionalOptions)

Specifies additional connection options.

Parameters:

additionalOptions - Specifies additional connection options.

Returns:

Returns a reference to this object so that method calls can be chained together.

additionalOptions

default S3ParquetSource.Builder additionalOptions(Consumer<S3DirectSourceAdditionalOptions.Builder> additionalOptions)

Specifies additional connection options.

This is a convenience method that creates an instance of the S3DirectSourceAdditionalOptions.Builder avoiding the need to create one manually via S3DirectSourceAdditionalOptions.builder().

When the Consumer completes, SdkBuilder.build() is called immediately and its result is passed to additionalOptions(S3DirectSourceAdditionalOptions).

Parameters:

additionalOptions - a consumer that will call methods on S3DirectSourceAdditionalOptions.Builder

Returns:

Returns a reference to this object so that method calls can be chained together.

See Also:

additionalOptions(S3DirectSourceAdditionalOptions)

outputSchemas

S3ParquetSource.Builder outputSchemas(Collection<GlueSchema> outputSchemas)

Specifies the data schema for the S3 Parquet source.

Parameters:

outputSchemas - Specifies the data schema for the S3 Parquet source.

Returns:

Returns a reference to this object so that method calls can be chained together.

outputSchemas

S3ParquetSource.Builder outputSchemas(GlueSchema... outputSchemas)

Specifies the data schema for the S3 Parquet source.

Parameters:

outputSchemas - Specifies the data schema for the S3 Parquet source.

Returns:

Returns a reference to this object so that method calls can be chained together.

outputSchemas

S3ParquetSource.Builder outputSchemas(Consumer<GlueSchema.Builder>... outputSchemas)

Specifies the data schema for the S3 Parquet source.

This is a convenience method that creates an instance of the GlueSchema.Builder avoiding the need to create one manually via GlueSchema.builder().

When the Consumer completes, SdkBuilder.build() is called immediately and its result is passed to #outputSchemas(List).

Parameters:

outputSchemas - a consumer that will call methods on GlueSchema.Builder

Returns:

Returns a reference to this object so that method calls can be chained together.

See Also:

#outputSchemas(java.util.Collection)