Package com.linkedin.dagli.text.token
-
Class Summary Class Description Tokens Tokenizes a string of natural text into its constituent tokens (words, punctuation, emoticons, etc.) using the best available tokenizer for each locale provided.
| Class | Description |
|---|---|
| Tokens |
Tokenizes a string of natural text into its constituent tokens (words, punctuation, emoticons, etc.) using the best
available tokenizer for each locale provided.
|