DynamicTFRecordDataset

Instance Constructors

new DynamicTFRecordDataset(filenames: Output, compressionType: CompressionType = NoCompression, bufferSize: Long = 256 * 1024, name: String = "TFRecordDataset")

filenames
Scalar or vector tensor containing the the name(s) of the file(s) to be read.
compressionType
Compression type for the file.
bufferSize
Number of bytes to buffer while reading from the file.
name
Name for this dataset.

Attributes
protected

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def createHandle(): Output

Creates a VARIANT scalar tensor representing this dataset.
Creates a VARIANT scalar tensor representing this dataset. This function adds ops to the current graph, that create the dataset resource.

Definition Classes
DynamicTFRecordDataset → Dataset
def createInitializableIterator(sharedName: String = "", name: String = "InitializableIterator"): InitializableIterator[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]

Creates an Iterator for enumerating the elements of this dataset.
Creates an Iterator for enumerating the elements of this dataset.
**Note:** The returned iterator will be in an uninitialized state. You must execute the InitializableIterator.initializer op before using it.
sharedName
If non-empty, then the constructed reader will be shared under the the provided name across multiple sessions that share the same devices (e.g., when using a remote server).
name
Name for the op created in relation to the iterator.
returns
Created iterator.

Definition Classes
Dataset
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
implicit val evData: Aux[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]

Definition Classes
Dataset
implicit val evFunctionInput: ArgType[Output]

Definition Classes
Dataset
implicit val evStructure: Aux[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]

Definition Classes
Dataset
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val name: String

Name for this dataset.
Name for this dataset.

Definition Classes
DynamicTFRecordDataset → Dataset
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def outputDataTypes: types.STRING

Returns the data types corresponding to each element of this dataset, matching the structure of the elements.
Returns the data types corresponding to each element of this dataset, matching the structure of the elements.

Definition Classes
DynamicTFRecordDataset → Dataset
def outputShapes: core.Shape

Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.
Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.

Definition Classes
DynamicTFRecordDataset → Dataset
def shard(numShards: Long, shardIndex: Long): Dataset[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]

Creates a dataset that includes only 1 / numShards of the elements of this dataset.
Creates a dataset that includes only 1 / numShards of the elements of this dataset.
This operator is very useful when running distributed training, as it allows each worker to read a unique subset of the dataset.
When reading a single input file, you can skip elements as follows:
```
tf.data.TFRecordDataset(inputFile)
  .shard(numWorkers, workerIndex)
  .repeat(numEpochs)
  .shuffle(shuffleBufferSize)
  .map(parserFn, numParallelCalls)
```
Important caveats:
- Be sure to shard before you use any randomizing operator (such as shuffle).
- Generally it is best if the shard operator is used early in the dataset pipeline. For example, when reading from a set of TensorFlow record files, shard before converting the dataset to input samples. This avoids reading every file on every worker. The following is an example of an efficient sharding strategy within a complete pipeline:
```
tf.data.listFiles(pattern)
  .shard(numWorkers, workerIndex)
  .repeat(numEpochs)
  .shuffle(shuffleBufferSize)
  .repeat()
  .interleave(tf.data.TFRecordDataset, cycleLength = numReaders, blockLength = 1)
  .map(parserFn, numParallelCalls)
```
numShards
Number of shards to use.
shardIndex
Index of the shard to obtain.
returns
Created (sharded) dataset.

Definition Classes
Dataset
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Dataset → AnyRef → Any
def transform[TT, TO, TD, TS](transformFn: (Dataset[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]) ⇒ Dataset[TT, TO, TD, TS])(implicit evStructure: Aux[TT, TO, TD, TS], evT: Aux[TT, TO, TD, TS], evFunctionInputT: ArgType[TO]): Dataset[TT, TO, TD, TS]

Applies a transformation function to this dataset.
Applies a transformation function to this dataset.
transform() enables chaining of custom dataset transformations, which are represented as functions that take one dataset argument and return a transformed dataset.
transformFn
Dataset transformation function.
returns
Transformed dataset.

Definition Classes
Dataset
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object DynamicTFRecordDataset | package data

class DynamicTFRecordDataset extends Dataset[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]

Instance Constructors

new DynamicTFRecordDataset(filenames: Output, compressionType: CompressionType = NoCompression, bufferSize: Long = 256 * 1024, name: String = "TFRecordDataset")

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

def createHandle(): Output

def createInitializableIterator(sharedName: String = "", name: String = "InitializableIterator"): InitializableIterator[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

implicit val evData: Aux[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]

implicit val evFunctionInput: ArgType[Output]

implicit val evStructure: Aux[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

val name: String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def outputDataTypes: types.STRING

def outputShapes: core.Shape

def shard(numShards: Long, shardIndex: Long): Dataset[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def transform[TT, TO, TD, TS](transformFn: (Dataset[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]) ⇒ Dataset[TT, TO, TD, TS])(implicit evStructure: Aux[TT, TO, TD, TS], evT: Aux[TT, TO, TD, TS], evFunctionInputT: ArgType[TO]): Dataset[TT, TO, TD, TS]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Dataset[tensors.Tensor[types.STRING], Output, types.STRING, core.Shape]

Inherited from AnyRef

Inherited from Any

Ungrouped