LookupTableTextFileInitializer

Lookup table initializer that uses a text file.

This initializer assigns one entry in the table for each line in the file. The key and value types of the table to initialize are given by keysDataType and valuesDataType.

The key and value content to extract from each line is specified by keysExtractor and valuesExtractor:

TextFileLineNumber: Use the line number (starting at zero -- expects INT64 data type).
TextFileWholeLine: Use the whole line content.
TextFileColumn(i): Use the ith element of the split line based on delimiter.

For example if we have a file with the following content:

emerson 10
lake 20
palmer 30

The following code creates an op that initializes a table with the first column as keys and second column as values:

val table = HashTable(LookupTableTextFileInitializer(
  "text.txt", STRING, INT64, TextFileColumn(0), TextFileColumn(1), " "))

Similarly to initialize the whole line as keys and the line number as values:

val table = HashTable(LookupTableTextFileInitializer(
  "text.txt", STRING, INT64, TextFileWholeLine, TextFileLineNumber, " "))

Linear Supertypes

LookupTableInitializer, AnyRef, Any

Instance Constructors

new LookupTableTextFileInitializer(filename: Output, keysDataType: types.DataType, valuesDataType: types.DataType, keysExtractor: TextFileFieldExtractor, valuesExtractor: TextFileFieldExtractor, delimiter: String = "\t", vocabularySize: Int = 1)

filename
Scalar STRING tensor containing the filename of the text file to be used for initialization. The path must be accessible from wherever the graph is initialized (e.g., trainer or evaluation workers).
keysDataType
Data type of the table keys.
valuesDataType
Data type of the table values.
keysExtractor
Text file field extractor to use for the keys (e.g., TextFileLineNumber).
valuesExtractor
Text file field extractor to use for the values (e.g., TextFileWholeLine).
delimiter
Delimiter to use in case a TextFileColumn extractor is being used.
vocabularySize
Number of elements in the file, if known. If not known, set to -1 (the default value).

Attributes
protected

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val delimiter: String

Delimiter to use in case a TextFileColumn extractor is being used.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
val filename: Output

Scalar STRING tensor containing the filename of the text file to be used for initialization.
Scalar STRING tensor containing the filename of the text file to be used for initialization. The path must be accessible from wherever the graph is initialized (e.g., trainer or evaluation workers).
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
def initialize(table: InitializableLookupTable, name: String = "LookupTableTextFileInitialize"): Op

Creates and returns an op that initializes the provided table.
Creates and returns an op that initializes the provided table.
table
Table to initialize.
returns
Created initialization op for table.

Definition Classes
LookupTableTextFileInitializer → LookupTableInitializer
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val keysDataType: types.DataType

Data type of the table keys.
Data type of the table keys.

Definition Classes
LookupTableTextFileInitializer → LookupTableInitializer
val keysExtractor: TextFileFieldExtractor

Text file field extractor to use for the keys (e.g., TextFileLineNumber).
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
val valuesDataType: types.DataType

Data type of the table values.
Data type of the table values.

Definition Classes
LookupTableTextFileInitializer → LookupTableInitializer
val valuesExtractor: TextFileFieldExtractor

Text file field extractor to use for the values (e.g., TextFileWholeLine).
val vocabularySize: Int

Number of elements in the file, if known.
Number of elements in the file, if known. If not known, set to -1 (the default value).
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object LookupTableTextFileInitializer | package lookup

class LookupTableTextFileInitializer extends LookupTableInitializer

Instance Constructors

new LookupTableTextFileInitializer(filename: Output, keysDataType: types.DataType, valuesDataType: types.DataType, keysExtractor: TextFileFieldExtractor, valuesExtractor: TextFileFieldExtractor, delimiter: String = "\t", vocabularySize: Int = 1)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

val delimiter: String

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

val filename: Output

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

def initialize(table: InitializableLookupTable, name: String = "LookupTableTextFileInitialize"): Op

final def isInstanceOf[T0]: Boolean

val keysDataType: types.DataType

val keysExtractor: TextFileFieldExtractor

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

val valuesDataType: types.DataType

val valuesExtractor: TextFileFieldExtractor

val vocabularySize: Int

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from LookupTableInitializer

Inherited from AnyRef

Inherited from Any

Ungrouped