software.amazon.awssdk.services.glue.model

Class S3ParquetSource

java.lang.Object

software.amazon.awssdk.services.glue.model.S3ParquetSource

All Implemented Interfaces:

Serializable, SdkPojo, ToCopyableBuilder<S3ParquetSource.Builder,S3ParquetSource>

@Generated(value="software.amazon.awssdk:codegen")
public final class S3ParquetSource
extends Object
implements SdkPojo, Serializable, ToCopyableBuilder<S3ParquetSource.Builder,S3ParquetSource>

Specifies an Apache Parquet data store stored in Amazon S3.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	S3ParquetSource.Builder

Method Summary

Modifier and Type	Method and Description
S3DirectSourceAdditionalOptions	additionalOptions() Specifies additional connection options.
static S3ParquetSource.Builder	builder()
ParquetCompressionType	compressionType() Specifies how the data is compressed.
String	compressionTypeAsString() Specifies how the data is compressed.
boolean	equals(Object obj)
boolean	equalsBySdkFields(Object obj)
List<String>	exclusions() A string containing a JSON list of Unix-style glob patterns to exclude.
<T> Optional<T>	getValueForField(String fieldName, Class<T> clazz)
String	groupFiles() Grouping files is turned on by default when the input contains more than 50,000 files.
String	groupSize() The target group size in bytes.
boolean	hasExclusions() For responses, this returns true if the service returned a value for the Exclusions property.
int	hashCode()
boolean	hasOutputSchemas() For responses, this returns true if the service returned a value for the OutputSchemas property.
boolean	hasPaths() For responses, this returns true if the service returned a value for the Paths property.
Integer	maxBand() This option controls the duration in milliseconds after which the s3 listing is likely to be consistent.
Integer	maxFilesInBand() This option specifies the maximum number of files to save from the last maxBand seconds.
String	name() The name of the data store.
List<GlueSchema>	outputSchemas() Specifies the data schema for the S3 Parquet source.
List<String>	paths() A list of the Amazon S3 paths to read from.
Boolean	recurse() If set to true, recursively reads files in all subdirectories under the specified paths.
List<SdkField<?>>	sdkFields()
static Class<? extends S3ParquetSource.Builder>	serializableBuilderClass()
S3ParquetSource.Builder	toBuilder()
String	toString() Returns a string representation of this object.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface software.amazon.awssdk.utils.builder.ToCopyableBuilder

copy

Method Detail

name

public final String name()

The name of the data store.

Returns:

The name of the data store.

hasPaths

public final boolean hasPaths()

For responses, this returns true if the service returned a value for the Paths property. This DOES NOT check that the value is non-empty (for which, you should check the isEmpty() method on the property). This is useful because the SDK will never return a null collection or map, but you may need to differentiate between the service returning nothing (or null) and the service returning an empty collection or map. For requests, this returns true if a value for the property was specified in the request builder, and false if a value was not specified.

paths

public final List<String> paths()

A list of the Amazon S3 paths to read from.

Attempts to modify the collection returned by this method will result in an UnsupportedOperationException.

This method will never return null. If you would like to know whether the service returned this field (so that you can differentiate between null and empty), you can use the hasPaths() method.

Returns:

A list of the Amazon S3 paths to read from.

compressionType

public final ParquetCompressionType compressionType()

Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

If the service returns an enum value that is not available in the current SDK version, compressionType will return ParquetCompressionType.UNKNOWN_TO_SDK_VERSION. The raw value returned by the service is available from compressionTypeAsString().

Returns:

Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

See Also:

ParquetCompressionType

compressionTypeAsString

public final String compressionTypeAsString()

Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

If the service returns an enum value that is not available in the current SDK version, compressionType will return ParquetCompressionType.UNKNOWN_TO_SDK_VERSION. The raw value returned by the service is available from compressionTypeAsString().

Returns:

Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

See Also:

ParquetCompressionType

hasExclusions

public final boolean hasExclusions()

For responses, this returns true if the service returned a value for the Exclusions property. This DOES NOT check that the value is non-empty (for which, you should check the isEmpty() method on the property). This is useful because the SDK will never return a null collection or map, but you may need to differentiate between the service returning nothing (or null) and the service returning an empty collection or map. For requests, this returns true if a value for the property was specified in the request builder, and false if a value was not specified.

exclusions

public final List<String> exclusions()

A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.

Attempts to modify the collection returned by this method will result in an UnsupportedOperationException.

This method will never return null. If you would like to know whether the service returned this field (so that you can differentiate between null and empty), you can use the hasExclusions() method.

Returns:

A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.

groupSize

public final String groupSize()

The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles" must be set to "inPartition" for this to take effect.

Returns:

The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles" must be set to "inPartition" for this to take effect.

groupFiles

public final String groupFiles()

Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none".

Returns:

Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none".

recurse

public final Boolean recurse()

If set to true, recursively reads files in all subdirectories under the specified paths.

Returns:

If set to true, recursively reads files in all subdirectories under the specified paths.

maxBand

public final Integer maxBand()

This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.

Returns:

This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.

maxFilesInBand

public final Integer maxFilesInBand()

This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.

Returns:

This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.

additionalOptions

public final S3DirectSourceAdditionalOptions additionalOptions()

Specifies additional connection options.

Returns:

Specifies additional connection options.

hasOutputSchemas

public final boolean hasOutputSchemas()

For responses, this returns true if the service returned a value for the OutputSchemas property. This DOES NOT check that the value is non-empty (for which, you should check the isEmpty() method on the property). This is useful because the SDK will never return a null collection or map, but you may need to differentiate between the service returning nothing (or null) and the service returning an empty collection or map. For requests, this returns true if a value for the property was specified in the request builder, and false if a value was not specified.

outputSchemas

public final List<GlueSchema> outputSchemas()

Specifies the data schema for the S3 Parquet source.

Attempts to modify the collection returned by this method will result in an UnsupportedOperationException.

This method will never return null. If you would like to know whether the service returned this field (so that you can differentiate between null and empty), you can use the hasOutputSchemas() method.

Returns:

Specifies the data schema for the S3 Parquet source.

toBuilder

public S3ParquetSource.Builder toBuilder()

Specified by:

toBuilder in interface ToCopyableBuilder<S3ParquetSource.Builder,S3ParquetSource>

builder

public static S3ParquetSource.Builder builder()

serializableBuilderClass

public static Class<? extends S3ParquetSource.Builder> serializableBuilderClass()

hashCode

public final int hashCode()

Overrides:

hashCode in class Object

equals

public final boolean equals(Object obj)

Overrides:

equals in class Object

equalsBySdkFields

public final boolean equalsBySdkFields(Object obj)

Specified by:

equalsBySdkFields in interface SdkPojo

toString

public final String toString()

Returns a string representation of this object. This is useful for testing and debugging. Sensitive data will be redacted from this string using a placeholder value.

Overrides:

toString in class Object

getValueForField

public final <T> Optional<T> getValueForField(String fieldName,
                                              Class<T> clazz)

sdkFields

public final List<SdkField<?>> sdkFields()

Specified by:

sdkFields in interface SdkPojo