lookup

Type Members

1. sealed trait HashSpecification extends AnyRef

   

   Hash specification for use with IDLookupTableWithHashBuckets.

2. case class HashTable(initializer: LookupTableInitializer, defaultValue: Output, container: String = "", sharedName: String = "", useNodeNameSharing: Boolean = false, name: String = "HashTable") extends InitializableLookupTable with Product with Serializable

   

   Generic hash table implementation for lookup tables.

   Generic hash table implementation for lookup tables.

   The constructor creates a hash table, specifying the type of its keys and values. Before using the table the caller will have to initialize it. After initialization the table will be immutable.

   Example usage:

   val table = HashTable(LookupTableTensorInitializer(keys, values), -1)
   val output = table.lookup(input)
   // Can now run evaluate the `output` tensor after executing the table initializer.

   initializer

   Lookup table initializer to use.

   defaultValue

   Default value to use if a key is missing from the table.

   container

   If non-empty, the created table is placed in the given container. Otherwise, a default container is used.

   sharedName

   If non-empty, the created table is named in the given bucket with this shared name. Otherwise, the op name is used, instead.

   useNodeNameSharing

   If set to true and sharedName is empty, the table is shared using the node name.

   name

   Name for the created table.

3. class IDLookupTableWithHashBuckets extends LookupTable

   

   String to ID lookup table wrapper that assigns out-of-vocabulary keys to buckets.

   String to ID lookup table wrapper that assigns out-of-vocabulary keys to buckets.

   For example, if an instance of IDLookupTableWithHashBuckets is initialized with a string-to-ID table that maps:

   emerson -> 0
   lake -> 1
   palmer -> 2

   The IDLookupTableWithHashBuckets object will perform the following mapping:

   emerson -> 0
   lake -> 1
   palmer -> 2
   <other term> -> bucket ID between 3 and 3 + numOOVBuckets - 1, calculated by hash(<term>) % numOOVBuckets + vocabularySize

   If the input tensor is ["emerson", "lake", "palmer", "king", "crimson"], the lookup result is [0, 1, 2, 4, 7]. If table is null, only out-of-vocabulary buckets are used.

   Example usage:

   val numOOVBuckets = 3
   val input = Tensor("emerson", "lake", "palmer", "king", "crimson")
   val table = IDLookupTableWithHashBuckets(
     HashTable(LookupTableTextFileInitializer(filename), defaultValue), numOOVBuckets)
   val output = table.lookup(input)

   The hash function used for generating out-of-vocabulary buckets ID is defined by hashSpecification.

4. abstract class InitializableLookupTable extends LookupTable

   

   Initializable lookup table that is constructed from an existing lookup table handle.

   Initializable lookup table that is constructed from an existing lookup table handle.

   Note that even though the caller needs to provide an initializer for this table, the caller also needs to make sure to execute the initialization op.

5. abstract class LookupTable extends AnyRef

   

   Lookup table that persists across different session runs.

6. abstract class LookupTableInitializer extends AnyRef

   

   Lookup table initializer.

7. class LookupTableTensorInitializer extends LookupTableInitializer

   

   Lookup table initializer that uses the provided tensors (containing keys and corresponding values) for initializing a lookup table.

8. class LookupTableTextFileInitializer extends LookupTableInitializer

   

   Lookup table initializer that uses a text file.

   Lookup table initializer that uses a text file.

   This initializer assigns one entry in the table for each line in the file. The key and value types of the table to initialize are given by keysDataType and valuesDataType.

   The key and value content to extract from each line is specified by keysExtractor and valuesExtractor:

   • TextFileLineNumber: Use the line number (starting at zero -- expects INT64 data type).
   • TextFileWholeLine: Use the whole line content.
   • TextFileColumn(i): Use the ith element of the split line based on delimiter.

   For example if we have a file with the following content:

   emerson 10
   lake 20
   palmer 30

   The following code creates an op that initializes a table with the first column as keys and second column as values:

   val table = HashTable(LookupTableTextFileInitializer(
     "text.txt", STRING, INT64, TextFileColumn(0), TextFileColumn(1), " "))

   Similarly to initialize the whole line as keys and the line number as values:

   val table = HashTable(LookupTableTextFileInitializer(
     "text.txt", STRING, INT64, TextFileWholeLine, TextFileLineNumber, " "))

9. case class STRONG_HASH(key1: Long, key2: Long) extends HashSpecification with Product with Serializable

   

10. case class TextFileColumn(index: Int) extends TextFileFieldExtractor with Product with Serializable

    

    Text file field extractor that extracts a column from a line as a field.

    Text file field extractor that extracts a column from a line as a field.

    index

    Column index.

11. sealed trait TextFileFieldExtractor extends AnyRef

    

    Represents a field extractor from a text file.