org.platanios.tensorflow.api.ops.io.data
FixedLengthRecordDataset
Related Doc:
package data
Scalar or vector tensor containing the the name(s) of the file(s) to be read.
Number of bytes in the record.
Number of bytes in the header (i.e., the number of bytes to skip at the beginning of a file).
Number of bytes in the footer (i.e., the number of bytes to skip at the end of a file).
Number of bytes to buffer while reading from the file.
Name for this dataset.
Number of bytes to buffer while reading from the file.
Creates a VARIANT scalar tensor representing this dataset.
Creates a VARIANT scalar tensor representing this dataset. This function adds ops to the current graph, that
create the dataset resource.
Creates an Iterator for enumerating the elements of this dataset.
Creates an Iterator for enumerating the elements of this dataset.
**Note:** The returned iterator will be in an uninitialized state. You must execute the InitializableIterator.initializer op before using it.
If non-empty, then the constructed reader will be shared under the the provided name across multiple sessions that share the same devices (e.g., when using a remote server).
Name for the op created in relation to the iterator.
Created iterator.
Scalar or vector tensor containing the the name(s) of the file(s) to be read.
Number of bytes in the footer (i.e., the number of bytes to skip at the end of a file).
Number of bytes in the header (i.e., the number of bytes to skip at the beginning of a file).
Name for this dataset.
Name for this dataset.
Returns the data types corresponding to each element of this dataset, matching the structure of the elements.
Returns the data types corresponding to each element of this dataset, matching the structure of the elements.
Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.
Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.
Number of bytes in the record.
Creates a dataset that includes only 1 / numShards of the elements of this dataset.
Creates a dataset that includes only 1 / numShards of the elements of this dataset.
This operator is very useful when running distributed training, as it allows each worker to read a unique subset of the dataset.
When reading a single input file, you can skip elements as follows:
tf.data.TFRecordDataset(inputFile) .shard(numWorkers, workerIndex) .repeat(numEpochs) .shuffle(shuffleBufferSize) .map(parserFn, numParallelCalls)
Important caveats:
tf.data.listFiles(pattern)
.shard(numWorkers, workerIndex)
.repeat(numEpochs)
.shuffle(shuffleBufferSize)
.repeat()
.interleave(tf.data.TFRecordDataset, cycleLength = numReaders, blockLength = 1)
.map(parserFn, numParallelCalls)Number of shards to use.
Index of the shard to obtain.
Created (sharded) dataset.
Applies a transformation function to this dataset.
Applies a transformation function to this dataset.
transform() enables chaining of custom dataset transformations, which are represented as functions that take one
dataset argument and return a transformed dataset.
Dataset transformation function.
Transformed dataset.
Dataset with elements read from binary files.
Scalar or vector tensor containing the the name(s) of the file(s) to be read.
Number of bytes in the record.
Number of bytes in the header (i.e., the number of bytes to skip at the beginning of a file).
Number of bytes in the footer (i.e., the number of bytes to skip at the end of a file).
Number of bytes to buffer while reading from the file.
Name for this dataset.