hyperparameter that gets multiplied once per backoff
An estimator has a fit method which emits a Transformer.
An estimator has a fit method which emits a Transformer.
Input data.
A Transformer which can be called on new data.
the pre-computed unigram counts of the training corpus
Constructs a pipeline from a single estimator and training data.
Constructs a pipeline from a single estimator and training data.
Equivalent to Pipeline() andThen (estimator, data)
The training data
Estimates a Stupid Backoff ngram language model, which was introduced in the following paper:
Brants, Thorsten, et al. "Large language models in machine translation." 2007.
The results are scores indicating likeliness of each ngram, but they are not normalized probabilities. The score for an n-gram is defined recursively:
S(w_i | w_{i - n + 1}{i - 1}) := if numerator > 0: freq(w_{i - n + 1}i) / freq(w_{i - n + 1}{i - 1}) otherwise: \alpha * S(w_i | w_{i - n + 2}{i - 1})
S(w_i) := freq(w_i) / N, where N is the total number of tokens in training corpus.
the pre-computed unigram counts of the training corpus
hyperparameter that gets multiplied once per backoff