Package opennlp.tools.tokenize
Class TokenizerModel
- java.lang.Object
-
- opennlp.tools.util.model.BaseModel
-
- opennlp.tools.tokenize.TokenizerModel
-
- All Implemented Interfaces:
java.io.Serializable
,ArtifactProvider
public final class TokenizerModel extends BaseModel
TheTokenizerModel
is the model used by a learnableTokenizer
.- See Also:
TokenizerME
, Serialized Form
-
-
Field Summary
-
Fields inherited from class opennlp.tools.util.model.BaseModel
TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY
-
-
Constructor Summary
Constructors Constructor Description TokenizerModel(java.io.File modelFile)
Initializes the current instance.TokenizerModel(java.io.InputStream in)
Initializes the current instance.TokenizerModel(java.net.URL modelURL)
Initializes the current instance.TokenizerModel(java.nio.file.Path modelPath)
TokenizerModel(MaxentModel tokenizerModel, java.util.Map<java.lang.String,java.lang.String> manifestInfoEntries, TokenizerFactory tokenizerFactory)
Initializes the current instance.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description Dictionary
getAbbreviations()
TokenizerFactory
getFactory()
MaxentModel
getMaxentModel()
boolean
useAlphaNumericOptimization()
-
Methods inherited from class opennlp.tools.util.model.BaseModel
getArtifact, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, serialize, serialize
-
-
-
-
Constructor Detail
-
TokenizerModel
public TokenizerModel(MaxentModel tokenizerModel, java.util.Map<java.lang.String,java.lang.String> manifestInfoEntries, TokenizerFactory tokenizerFactory)
Initializes the current instance.- Parameters:
tokenizerModel
- the modelmanifestInfoEntries
- the manifesttokenizerFactory
- the factory
-
TokenizerModel
public TokenizerModel(java.io.InputStream in) throws java.io.IOException
Initializes the current instance.- Parameters:
in
- the Input Stream to load the model from- Throws:
java.io.IOException
- if reading from the stream fails in anywayInvalidFormatException
- if the stream doesn't have the expected format
-
TokenizerModel
public TokenizerModel(java.io.File modelFile) throws java.io.IOException
Initializes the current instance.- Parameters:
modelFile
- the file containing the tokenizer model- Throws:
java.io.IOException
- if reading from the stream fails in anyway
-
TokenizerModel
public TokenizerModel(java.nio.file.Path modelPath) throws java.io.IOException
- Throws:
java.io.IOException
-
TokenizerModel
public TokenizerModel(java.net.URL modelURL) throws java.io.IOException
Initializes the current instance.- Parameters:
modelURL
- the URL pointing to the tokenizer model- Throws:
java.io.IOException
- if reading from the stream fails in anyway
-
-
Method Detail
-
getFactory
public TokenizerFactory getFactory()
-
getMaxentModel
public MaxentModel getMaxentModel()
-
getAbbreviations
public Dictionary getAbbreviations()
-
useAlphaNumericOptimization
public boolean useAlphaNumericOptimization()
-
-