com.intel.analytics.zoo.feature.text
LocalTextSet
Related Doc:
package text
Generate BigDL Sample.
Generate BigDL Sample. Need to word2idx first. See TextFeatureToSample for more details.
Generate wordIndex map based on sorted word frequencies in descending order.
Generate wordIndex map based on sorted word frequencies in descending order. Return the result map, which will also be stored in 'wordIndex'. Make sure you call this after tokenize. Otherwise you will get an exception. See word2idx for more details.
Get the word index map of this TextSet.
Get the word index map of this TextSet. If the TextSet hasn't been transformed from word to index, null will be returned.
Whether it is a DistributedTextSet.
Whether it is a DistributedTextSet.
Whether it is a LocalTextSet.
Whether it is a LocalTextSet.
Load the wordIndex map which was saved after the training, so that this TextSet can directly use this wordIndex during inference.
Load the wordIndex map which was saved after the training, so that this TextSet can directly use this wordIndex during inference. Each separate line should be "word id".
Note that after calling loadWordIndex, you do not need to specify any argument when calling
word2idx in the preprocessing pipeline as now you are using exactly the loaded wordIndex for
transformation.
For LocalTextSet, load txt from a local file system. For DistributedTextSet, load txt from a local or distributed file system (such as HDFS).
The path to the text file.
Do normalization on tokens.
Do normalization on tokens. Need to tokenize first. See Normalizer for more details.
Randomly split into array of TextSet with provided weights.
Randomly split into array of TextSet with provided weights. Only available for DistributedTextSet for now.
Array of Double indicating the split portions.
Save wordIndex map to text file, which can be used for future inference.
Save wordIndex map to text file, which can be used for future inference. Each separate line will be "word id".
For LocalTextSet, save txt to a local file system. For DistributedTextSet, save txt to a local or distributed file system (such as HDFS).
The path to the text file.
Assign a wordIndex map for this TextSet to use during word2idx.
Assign a wordIndex map for this TextSet to use during word2idx.
If you load the wordIndex from the saved file, you are recommended to use loadWordIndex
directly.
Map of each word (String) and its index (integer).
Shape the sequence of indices to a fixed length.
Shape the sequence of indices to a fixed length. Need to word2idx first. See SequenceShaper for more details.
Convert TextSet to DataSet of Sample.
Convert TextSet to DataSet of Sample.
Convert to a DistributedTextSet.
Convert to a DistributedTextSet.
Need to specify SparkContext to convert a LocalTextSet to a DistributedTextSet. In this case, you may also want to specify partitionNum, the default of which is 4.
Convert to a LocalTextSet.
Convert to a LocalTextSet.
Do tokenization on original text.
Transform from one TextSet to another.
Transform from one TextSet to another.
Map word tokens to indices.
Map word tokens to indices. Important: Take care that this method behaves a bit differently for training and inference.
During the training, you need to generate a new wordIndex map according to the texts you are
dealing with. Thus this method will first do the map generation and then convert words to
indices based on the generated map.
You can specify the following arguments which pose some constraints when generating the map.
In the result map, index will start from 1 and corresponds to the occurrence frequency of
each word sorted in descending order.
Here we adopt the convention that index 0 will be reserved for unknown words.
After word2idx, you can get the generated wordIndex map by calling 'getWordIndex'.
Also, you can call saveWordIndex to save this wordIndex map to be used in future training.
Non-negative integer. Remove the topN words with highest frequencies in the case where those are treated as stopwords. Default is 0, namely remove nothing.
Integer. The maximum number of words to be taken into consideration. Default is -1, namely all words will be considered. Otherwise, it should be a positive integer.
Positive integer. Only those words with frequency >= minFreq will be taken into consideration. Default is 1, namely all words that occur will be considered.
Existing map of word index if any. Default is null and in this case a new
map with index starting from 1 will be generated.
If not null, then the generated map will preserve the word index in
existingMap and assign subsequent indices to new words.
---------------------------------------Inference--------------------------------------------
During the inference, you are supposed to use exactly the same wordIndex map as in the
training stage instead of generating a new one.
Thus please be aware that you do not need to specify any of the above arguments.
You need to call loadWordIndex or setWordIndex beforehand for map loading.
Need to tokenize first.
See WordIndexer for more details.
LocalTextSet is comprised of array of TextFeature.