public static interface S3DataSource.Builder extends SdkPojo, CopyableBuilder<S3DataSource.Builder,S3DataSource>
| Modifier and Type | Method and Description |
|---|---|
S3DataSource.Builder |
attributeNames(Collection<String> attributeNames)
A list of one or more attribute names to use that are found in a specified augmented manifest file.
|
S3DataSource.Builder |
attributeNames(String... attributeNames)
A list of one or more attribute names to use that are found in a specified augmented manifest file.
|
S3DataSource.Builder |
s3DataDistributionType(S3DataDistribution s3DataDistributionType)
If you want Amazon SageMaker to replicate the entire dataset on each ML compute instance that is launched for
model training, specify
FullyReplicated. |
S3DataSource.Builder |
s3DataDistributionType(String s3DataDistributionType)
If you want Amazon SageMaker to replicate the entire dataset on each ML compute instance that is launched for
model training, specify
FullyReplicated. |
S3DataSource.Builder |
s3DataType(S3DataType s3DataType)
If you choose
S3Prefix, S3Uri identifies a key name prefix. |
S3DataSource.Builder |
s3DataType(String s3DataType)
If you choose
S3Prefix, S3Uri identifies a key name prefix. |
S3DataSource.Builder |
s3Uri(String s3Uri)
Depending on the value specified for the
S3DataType, identifies either a key name prefix or a
manifest. |
equalsBySdkFields, sdkFieldscopyapplyMutation, buildS3DataSource.Builder s3DataType(String s3DataType)
If you choose S3Prefix, S3Uri identifies a key name prefix. Amazon SageMaker uses
all objects that match the specified key name prefix for model training.
If you choose ManifestFile, S3Uri identifies an object that is a manifest file
containing a list of object keys that you want Amazon SageMaker to use for model training.
If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest
file in JSON lines format. This file contains the data you want to use for model training.
AugmentedManifestFile can only be used if the Channel's input mode is Pipe.
s3DataType - If you choose S3Prefix, S3Uri identifies a key name prefix. Amazon SageMaker
uses all objects that match the specified key name prefix for model training.
If you choose ManifestFile, S3Uri identifies an object that is a manifest
file containing a list of object keys that you want Amazon SageMaker to use for model training.
If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented
manifest file in JSON lines format. This file contains the data you want to use for model training.
AugmentedManifestFile can only be used if the Channel's input mode is Pipe.
S3DataType,
S3DataTypeS3DataSource.Builder s3DataType(S3DataType s3DataType)
If you choose S3Prefix, S3Uri identifies a key name prefix. Amazon SageMaker uses
all objects that match the specified key name prefix for model training.
If you choose ManifestFile, S3Uri identifies an object that is a manifest file
containing a list of object keys that you want Amazon SageMaker to use for model training.
If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented manifest
file in JSON lines format. This file contains the data you want to use for model training.
AugmentedManifestFile can only be used if the Channel's input mode is Pipe.
s3DataType - If you choose S3Prefix, S3Uri identifies a key name prefix. Amazon SageMaker
uses all objects that match the specified key name prefix for model training.
If you choose ManifestFile, S3Uri identifies an object that is a manifest
file containing a list of object keys that you want Amazon SageMaker to use for model training.
If you choose AugmentedManifestFile, S3Uri identifies an object that is an augmented
manifest file in JSON lines format. This file contains the data you want to use for model training.
AugmentedManifestFile can only be used if the Channel's input mode is Pipe.
S3DataType,
S3DataTypeS3DataSource.Builder s3Uri(String s3Uri)
Depending on the value specified for the S3DataType, identifies either a key name prefix or a
manifest. For example:
A key name prefix might look like this: s3://bucketname/exampleprefix.
A manifest might look like this: s3://bucketname/example.manifest
The manifest is an S3 object which is a JSON file with the following format:
The preceding JSON matches the following s3Uris:
[ {"prefix": "s3://customer_bucket/some/prefix/"},
"relative/path/to/custdata-1",
"relative/path/custdata-2",
...
"relative/path/custdata-N"
]
The preceding JSON matches the following s3Uris:
s3://customer_bucket/some/prefix/relative/path/to/custdata-1
s3://customer_bucket/some/prefix/relative/path/custdata-2
...
s3://customer_bucket/some/prefix/relative/path/custdata-N
The complete set of s3uris in this manifest is the input data for the channel for this
datasource. The object that each s3uris points to must be readable by the IAM role that Amazon
SageMaker uses to perform tasks on your behalf.
s3Uri - Depending on the value specified for the S3DataType, identifies either a key name prefix
or a manifest. For example:
A key name prefix might look like this: s3://bucketname/exampleprefix.
A manifest might look like this: s3://bucketname/example.manifest
The manifest is an S3 object which is a JSON file with the following format:
The preceding JSON matches the following s3Uris:
[ {"prefix": "s3://customer_bucket/some/prefix/"},
"relative/path/to/custdata-1",
"relative/path/custdata-2",
...
"relative/path/custdata-N"
]
The preceding JSON matches the following s3Uris:
s3://customer_bucket/some/prefix/relative/path/to/custdata-1
s3://customer_bucket/some/prefix/relative/path/custdata-2
...
s3://customer_bucket/some/prefix/relative/path/custdata-N
The complete set of s3uris in this manifest is the input data for the channel for this
datasource. The object that each s3uris points to must be readable by the IAM role that
Amazon SageMaker uses to perform tasks on your behalf.
S3DataSource.Builder s3DataDistributionType(String s3DataDistributionType)
If you want Amazon SageMaker to replicate the entire dataset on each ML compute instance that is launched for
model training, specify FullyReplicated.
If you want Amazon SageMaker to replicate a subset of data on each ML compute instance that is launched for
model training, specify ShardedByS3Key. If there are n ML compute instances launched for
a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model
training on each machine uses only the subset of training data.
Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose
ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when
TrainingInputMode is set to File), this copies 1/n of the number of objects.
s3DataDistributionType - If you want Amazon SageMaker to replicate the entire dataset on each ML compute instance that is
launched for model training, specify FullyReplicated.
If you want Amazon SageMaker to replicate a subset of data on each ML compute instance that is
launched for model training, specify ShardedByS3Key. If there are n ML compute
instances launched for a training job, each instance gets approximately 1/n of the number of S3
objects. In this case, model training on each machine uses only the subset of training data.
Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose
ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume
(when TrainingInputMode is set to File), this copies 1/n of the
number of objects.
S3DataDistribution,
S3DataDistributionS3DataSource.Builder s3DataDistributionType(S3DataDistribution s3DataDistributionType)
If you want Amazon SageMaker to replicate the entire dataset on each ML compute instance that is launched for
model training, specify FullyReplicated.
If you want Amazon SageMaker to replicate a subset of data on each ML compute instance that is launched for
model training, specify ShardedByS3Key. If there are n ML compute instances launched for
a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model
training on each machine uses only the subset of training data.
Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose
ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume (when
TrainingInputMode is set to File), this copies 1/n of the number of objects.
s3DataDistributionType - If you want Amazon SageMaker to replicate the entire dataset on each ML compute instance that is
launched for model training, specify FullyReplicated.
If you want Amazon SageMaker to replicate a subset of data on each ML compute instance that is
launched for model training, specify ShardedByS3Key. If there are n ML compute
instances launched for a training job, each instance gets approximately 1/n of the number of S3
objects. In this case, model training on each machine uses only the subset of training data.
Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose
ShardedByS3Key. If the algorithm requires copying training data to the ML storage volume
(when TrainingInputMode is set to File), this copies 1/n of the
number of objects.
S3DataDistribution,
S3DataDistributionS3DataSource.Builder attributeNames(Collection<String> attributeNames)
A list of one or more attribute names to use that are found in a specified augmented manifest file.
attributeNames - A list of one or more attribute names to use that are found in a specified augmented manifest file.S3DataSource.Builder attributeNames(String... attributeNames)
A list of one or more attribute names to use that are found in a specified augmented manifest file.
attributeNames - A list of one or more attribute names to use that are found in a specified augmented manifest file.Copyright © 2020. All rights reserved.