org.platanios.tensorflow.api.ops.io.data
ParallelInterleaveDataset
Related Doc:
package data
Input dataset.
Mapping function.
Number of elements from the input dataset that will be processed concurrently.
Number of consecutive elements to produce from each input element before cycling to another input element.
If false, elements are produced in deterministic order. Otherwise, the
implementation is allowed, for the sake of expediency, to produce elements in a
non-deterministic order.
Number of elements each iterator being interleaved should buffer (similar to the
prefetch(...) transformation for each interleaved iterator).
Number of input elements to transform to iterators before they are needed for interleaving.
Name for this dataset.
Number of consecutive elements to produce from each input element before cycling to another input element.
Number of elements each iterator being interleaved should buffer (similar to the
prefetch(...) transformation for each interleaved iterator).
Creates a VARIANT scalar tensor representing this dataset.
Creates a VARIANT scalar tensor representing this dataset. This function adds ops to the current graph, that
create the dataset resource.
Creates an Iterator for enumerating the elements of this dataset.
Creates an Iterator for enumerating the elements of this dataset.
**Note:** The returned iterator will be in an uninitialized state. You must execute the InitializableIterator.initializer op before using it.
If non-empty, then the constructed reader will be shared under the the provided name across multiple sessions that share the same devices (e.g., when using a remote server).
Name for the op created in relation to the iterator.
Created iterator.
Number of elements from the input dataset that will be processed concurrently.
Mapping function.
Input dataset.
Name for this dataset.
Name for this dataset.
Returns the data types corresponding to each element of this dataset, matching the structure of the elements.
Returns the data types corresponding to each element of this dataset, matching the structure of the elements.
Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.
Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.
Number of input elements to transform to iterators before they are needed for interleaving.
Creates a dataset that includes only 1 / numShards of the elements of this dataset.
Creates a dataset that includes only 1 / numShards of the elements of this dataset.
This operator is very useful when running distributed training, as it allows each worker to read a unique subset of the dataset.
When reading a single input file, you can skip elements as follows:
tf.data.TFRecordDataset(inputFile) .shard(numWorkers, workerIndex) .repeat(numEpochs) .shuffle(shuffleBufferSize) .map(parserFn, numParallelCalls)
Important caveats:
tf.data.listFiles(pattern)
.shard(numWorkers, workerIndex)
.repeat(numEpochs)
.shuffle(shuffleBufferSize)
.repeat()
.interleave(tf.data.TFRecordDataset, cycleLength = numReaders, blockLength = 1)
.map(parserFn, numParallelCalls)Number of shards to use.
Index of the shard to obtain.
Created (sharded) dataset.
If false, elements are produced in deterministic order.
If false, elements are produced in deterministic order. Otherwise, the
implementation is allowed, for the sake of expediency, to produce elements in a
non-deterministic order.
Applies a transformation function to this dataset.
Applies a transformation function to this dataset.
transform() enables chaining of custom dataset transformations, which are represented as functions that take one
dataset argument and return a transformed dataset.
Dataset transformation function.
Transformed dataset.
Dataset that wraps the application of the
parallelInterleaveop.$OpDocDatasetParallelInterleave
Tensor type (i.e., nested structure of tensors).
Output type (i.e., nested structure of symbolic tensors).
Data type of the outputs (i.e., nested structure of TensorFlow data types).
Shape type of the outputs (i.e., nested structure of TensorFlow shapes).
Input dataset.
Mapping function.
Number of elements from the input dataset that will be processed concurrently.
Number of consecutive elements to produce from each input element before cycling to another input element.
If
false, elements are produced in deterministic order. Otherwise, the implementation is allowed, for the sake of expediency, to produce elements in a non-deterministic order.Number of elements each iterator being interleaved should buffer (similar to the
prefetch(...)transformation for each interleaved iterator).Number of input elements to transform to iterators before they are needed for interleaving.
Name for this dataset.