org.platanios.tensorflow.api.ops.io.data
Input dataset.
Mapping function.
Name for this dataset.
Creates a VARIANT
scalar tensor representing this dataset.
Creates a VARIANT
scalar tensor representing this dataset. This function adds ops to the current graph, that
create the dataset resource.
Creates an Iterator for enumerating the elements of this dataset.
Creates an Iterator for enumerating the elements of this dataset.
**Note:** The returned iterator will be in an uninitialized state. You must execute the InitializableIterator.initializer op before using it.
If non-empty, then the constructed reader will be shared under the the provided name across multiple sessions that share the same devices (e.g., when using a remote server).
Name for the op created in relation to the iterator.
Created iterator.
Mapping function.
Input dataset.
Name for this dataset.
Name for this dataset.
Returns the data types corresponding to each element of this dataset, matching the structure of the elements.
Returns the data types corresponding to each element of this dataset, matching the structure of the elements.
Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.
Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.
Creates a dataset that includes only 1 / numShards
of the elements of this dataset.
Creates a dataset that includes only 1 / numShards
of the elements of this dataset.
This operator is very useful when running distributed training, as it allows each worker to read a unique subset of the dataset.
When reading a single input file, you can skip elements as follows:
tf.data.TFRecordDataset(inputFile) .shard(numWorkers, workerIndex) .repeat(numEpochs) .shuffle(shuffleBufferSize) .map(parserFn, numParallelCalls)
Important caveats:
tf.data.listFiles(pattern)
.shard(numWorkers, workerIndex)
.repeat(numEpochs)
.shuffle(shuffleBufferSize)
.repeat()
.interleave(tf.data.TFRecordDataset, cycleLength = numReaders, blockLength = 1)
.map(parserFn, numParallelCalls)
Number of shards to use.
Index of the shard to obtain.
Created (sharded) dataset.
Applies a transformation function to this dataset.
Applies a transformation function to this dataset.
transform()
enables chaining of custom dataset transformations, which are represented as functions that take one
dataset argument and return a transformed dataset.
Dataset transformation function.
Transformed dataset.
Dataset that wraps the application of the
flatMap
op.$OpDocDatasetFlatMap
Tensor type (i.e., nested structure of tensors).
Output type (i.e., nested structure of symbolic tensors).
Data type of the outputs (i.e., nested structure of TensorFlow data types).
Shape type of the outputs (i.e., nested structure of TensorFlow shapes).
Input dataset.
Mapping function.
Name for this dataset.