BatchDataset

Instance Constructors

new BatchDataset(inputDataset: Dataset[T, O, D, S], batchSize: Long, name: String = "BatchDataset")

inputDataset
Input dataset.
batchSize
Batch size to use.
name
Name for this dataset.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
val batchSize: Long

Batch size to use.
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def createHandle(): Output

Creates a VARIANT scalar tensor representing this dataset.
Creates a VARIANT scalar tensor representing this dataset. This function adds ops to the current graph, that create the dataset resource.

Definition Classes
BatchDataset → Dataset
def createInitializableIterator(sharedName: String = "", name: String = "InitializableIterator"): InitializableIterator[T, O, D, S]

Creates an Iterator for enumerating the elements of this dataset.
Creates an Iterator for enumerating the elements of this dataset.
**Note:** The returned iterator will be in an uninitialized state. You must execute the InitializableIterator.initializer op before using it.
sharedName
If non-empty, then the constructed reader will be shared under the the provided name across multiple sessions that share the same devices (e.g., when using a remote server).
name
Name for the op created in relation to the iterator.
returns
Created iterator.

Definition Classes
Dataset
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
implicit val evData: Aux[T, O, D, S]

Definition Classes
Dataset
implicit val evFunctionInput: ArgType[O]

Definition Classes
Dataset
implicit val evStructure: Aux[T, O, D, S]

Definition Classes
Dataset
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
val inputDataset: Dataset[T, O, D, S]

Input dataset.
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val name: String

Name for this dataset.
Name for this dataset.

Definition Classes
BatchDataset → Dataset
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def outputDataTypes: D

Returns the data types corresponding to each element of this dataset, matching the structure of the elements.
Returns the data types corresponding to each element of this dataset, matching the structure of the elements.

Definition Classes
BatchDataset → Dataset
def outputShapes: S

Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.
Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.

Definition Classes
BatchDataset → Dataset
def shard(numShards: Long, shardIndex: Long): Dataset[T, O, D, S]

Creates a dataset that includes only 1 / numShards of the elements of this dataset.
Creates a dataset that includes only 1 / numShards of the elements of this dataset.
This operator is very useful when running distributed training, as it allows each worker to read a unique subset of the dataset.
When reading a single input file, you can skip elements as follows:
```
tf.data.TFRecordDataset(inputFile)
  .shard(numWorkers, workerIndex)
  .repeat(numEpochs)
  .shuffle(shuffleBufferSize)
  .map(parserFn, numParallelCalls)
```
Important caveats:
- Be sure to shard before you use any randomizing operator (such as shuffle).
- Generally it is best if the shard operator is used early in the dataset pipeline. For example, when reading from a set of TensorFlow record files, shard before converting the dataset to input samples. This avoids reading every file on every worker. The following is an example of an efficient sharding strategy within a complete pipeline:
```
tf.data.listFiles(pattern)
  .shard(numWorkers, workerIndex)
  .repeat(numEpochs)
  .shuffle(shuffleBufferSize)
  .repeat()
  .interleave(tf.data.TFRecordDataset, cycleLength = numReaders, blockLength = 1)
  .map(parserFn, numParallelCalls)
```
numShards
Number of shards to use.
shardIndex
Index of the shard to obtain.
returns
Created (sharded) dataset.

Definition Classes
Dataset
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Dataset → AnyRef → Any
def transform[TT, TO, TD, TS](transformFn: (Dataset[T, O, D, S]) ⇒ Dataset[TT, TO, TD, TS])(implicit evStructure: Aux[TT, TO, TD, TS], evT: Aux[TT, TO, TD, TS], evFunctionInputT: ArgType[TO]): Dataset[TT, TO, TD, TS]

Applies a transformation function to this dataset.
Applies a transformation function to this dataset.
transform() enables chaining of custom dataset transformations, which are represented as functions that take one dataset argument and return a transformed dataset.
transformFn
Dataset transformation function.
returns
Transformed dataset.

Definition Classes
Dataset
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object BatchDataset | package data

case class BatchDataset[T, O, D, S](inputDataset: Dataset[T, O, D, S], batchSize: Long, name: String = "BatchDataset") extends Dataset[T, O, D, S] with Product with Serializable

Instance Constructors

new BatchDataset(inputDataset: Dataset[T, O, D, S], batchSize: Long, name: String = "BatchDataset")

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

val batchSize: Long

def clone(): AnyRef

def createHandle(): Output

def createInitializableIterator(sharedName: String = "", name: String = "InitializableIterator"): InitializableIterator[T, O, D, S]

final def eq(arg0: AnyRef): Boolean

implicit val evData: Aux[T, O, D, S]

implicit val evFunctionInput: ArgType[O]

implicit val evStructure: Aux[T, O, D, S]

def finalize(): Unit

final def getClass(): Class[_]

val inputDataset: Dataset[T, O, D, S]

final def isInstanceOf[T0]: Boolean

val name: String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def outputDataTypes: D

def outputShapes: S

def shard(numShards: Long, shardIndex: Long): Dataset[T, O, D, S]

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def transform[TT, TO, TD, TS](transformFn: (Dataset[T, O, D, S]) ⇒ Dataset[TT, TO, TD, TS])(implicit evStructure: Aux[TT, TO, TD, TS], evT: Aux[TT, TO, TD, TS], evFunctionInputT: ArgType[TO]): Dataset[TT, TO, TD, TS]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Dataset[T, O, D, S]

Inherited from AnyRef

Inherited from Any

Ungrouped