InterleaveDataset

Instance Constructors

new InterleaveDataset(inputDataset: Dataset[T, O, D, S], function: (O) ⇒ Dataset[RT, RO, RD, RS], cycleLength: Output, blockLength: Output = 1, name: String = "InterleaveDataset")(implicit evData: Aux[T, O, D, S] = inputDataset.evData, evFunctionInput: ArgType[O] = inputDataset.evFunctionInput, evStructure: Aux[RT, RO, RD, RS], evRData: Aux[RT, RO, RD, RS], evFunctionOutput: ArgType[RO])

inputDataset
Input dataset.
function
Mapping function.
cycleLength
Number of elements from the input dataset that will be processed concurrently.
blockLength
Number of consecutive elements to produce from each input element before cycling to another input element.
name
Name for this dataset.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
val blockLength: Output

Number of consecutive elements to produce from each input element before cycling to another input element.
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def createHandle(): Output

Creates a VARIANT scalar tensor representing this dataset.
Creates a VARIANT scalar tensor representing this dataset. This function adds ops to the current graph, that create the dataset resource.

Definition Classes
InterleaveDataset → Dataset
def createInitializableIterator(sharedName: String = "", name: String = "InitializableIterator"): InitializableIterator[RT, RO, RD, RS]

Creates an Iterator for enumerating the elements of this dataset.
Creates an Iterator for enumerating the elements of this dataset.
**Note:** The returned iterator will be in an uninitialized state. You must execute the InitializableIterator.initializer op before using it.
sharedName
If non-empty, then the constructed reader will be shared under the the provided name across multiple sessions that share the same devices (e.g., when using a remote server).
name
Name for the op created in relation to the iterator.
returns
Created iterator.

Definition Classes
Dataset
val cycleLength: Output

Number of elements from the input dataset that will be processed concurrently.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
implicit val evData: Aux[RT, RO, RD, RS]

Definition Classes
Dataset
implicit val evFunctionInput: ArgType[RO]

Definition Classes
Dataset
implicit val evStructure: Aux[RT, RO, RD, RS]

Definition Classes
Dataset
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
val function: (O) ⇒ Dataset[RT, RO, RD, RS]

Mapping function.
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
val inputDataset: Dataset[T, O, D, S]

Input dataset.
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val name: String

Name for this dataset.
Name for this dataset.

Definition Classes
InterleaveDataset → Dataset
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def outputDataTypes: RD

Returns the data types corresponding to each element of this dataset, matching the structure of the elements.
Returns the data types corresponding to each element of this dataset, matching the structure of the elements.

Definition Classes
InterleaveDataset → Dataset
def outputShapes: RS

Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.
Returns the shapes corresponding to each element of this dataset, matching the structure of the elements.

Definition Classes
InterleaveDataset → Dataset
def shard(numShards: Long, shardIndex: Long): Dataset[RT, RO, RD, RS]

Creates a dataset that includes only 1 / numShards of the elements of this dataset.
Creates a dataset that includes only 1 / numShards of the elements of this dataset.
This operator is very useful when running distributed training, as it allows each worker to read a unique subset of the dataset.
When reading a single input file, you can skip elements as follows:
```
tf.data.TFRecordDataset(inputFile)
  .shard(numWorkers, workerIndex)
  .repeat(numEpochs)
  .shuffle(shuffleBufferSize)
  .map(parserFn, numParallelCalls)
```
Important caveats:
- Be sure to shard before you use any randomizing operator (such as shuffle).
- Generally it is best if the shard operator is used early in the dataset pipeline. For example, when reading from a set of TensorFlow record files, shard before converting the dataset to input samples. This avoids reading every file on every worker. The following is an example of an efficient sharding strategy within a complete pipeline:
```
tf.data.listFiles(pattern)
  .shard(numWorkers, workerIndex)
  .repeat(numEpochs)
  .shuffle(shuffleBufferSize)
  .repeat()
  .interleave(tf.data.TFRecordDataset, cycleLength = numReaders, blockLength = 1)
  .map(parserFn, numParallelCalls)
```
numShards
Number of shards to use.
shardIndex
Index of the shard to obtain.
returns
Created (sharded) dataset.

Definition Classes
Dataset
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Dataset → AnyRef → Any
def transform[TT, TO, TD, TS](transformFn: (Dataset[RT, RO, RD, RS]) ⇒ Dataset[TT, TO, TD, TS])(implicit evStructure: Aux[TT, TO, TD, TS], evT: Aux[TT, TO, TD, TS], evFunctionInputT: ArgType[TO]): Dataset[TT, TO, TD, TS]

Applies a transformation function to this dataset.
Applies a transformation function to this dataset.
transform() enables chaining of custom dataset transformations, which are represented as functions that take one dataset argument and return a transformed dataset.
transformFn
Dataset transformation function.
returns
Transformed dataset.

Definition Classes
Dataset
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object InterleaveDataset | package data

Instance Constructors

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

val blockLength: Output

def clone(): AnyRef

def createHandle(): Output

def createInitializableIterator(sharedName: String = "", name: String = "InitializableIterator"): InitializableIterator[RT, RO, RD, RS]

val cycleLength: Output

final def eq(arg0: AnyRef): Boolean

implicit val evData: Aux[RT, RO, RD, RS]

implicit val evFunctionInput: ArgType[RO]

implicit val evStructure: Aux[RT, RO, RD, RS]

def finalize(): Unit

val function: (O) ⇒ Dataset[RT, RO, RD, RS]

final def getClass(): Class[_]

val inputDataset: Dataset[T, O, D, S]

final def isInstanceOf[T0]: Boolean

val name: String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def outputDataTypes: RD

def outputShapes: RS

def shard(numShards: Long, shardIndex: Long): Dataset[RT, RO, RD, RS]

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def transform[TT, TO, TD, TS](transformFn: (Dataset[RT, RO, RD, RS]) ⇒ Dataset[TT, TO, TD, TS])(implicit evStructure: Aux[TT, TO, TD, TS], evT: Aux[TT, TO, TD, TS], evFunctionInputT: ArgType[TO]): Dataset[TT, TO, TD, TS]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Dataset[RT, RO, RD, RS]

Inherited from AnyRef

Inherited from Any

Ungrouped