Package ai.djl.sentencepiece

Contains classes to interface with the underlying SentencePiece.

Class Summary

Class	Description
SpTextEmbedding	A TextEmbedding in SentencePiece that do sentence tokenization and map tokens into indices.
SpTokenizer	SpTokenizer is a SentencePiece implementation of the Tokenizer interface that converts sentences into token.
SpVocabulary	SpVocabulary is a SentencePiece implementation of Vocabulary.