A B C D E F G I L P S T U
All Classes All Packages

A

ai.djl.sentencepiece - package ai.djl.sentencepiece

Contains classes to interface with the underlying SentencePiece.

ai.djl.sentencepiece.jni - package ai.djl.sentencepiece.jni

Contains classes to interface with the underlying SentencePiece Engine.

B

buildSentence(List<String>) - Method in class ai.djl.sentencepiece.SpTokenizer

C

close() - Method in class ai.djl.sentencepiece.SpTokenizer

contains(String) - Method in class ai.djl.sentencepiece.SpVocabulary

createSentencePieceProcessor() - Method in class ai.djl.sentencepiece.jni.SentencePieceLibrary

D

decode(long, int[]) - Method in class ai.djl.sentencepiece.jni.SentencePieceLibrary

deleteSentencePieceProcessor(long) - Method in class ai.djl.sentencepiece.jni.SentencePieceLibrary

detokenize(long, String[]) - Method in class ai.djl.sentencepiece.jni.SentencePieceLibrary

E

embedText(NDArray) - Method in class ai.djl.sentencepiece.SpTextEmbedding

encode(long, String) - Method in class ai.djl.sentencepiece.jni.SentencePieceLibrary

F

from(SpTokenizer) - Static method in class ai.djl.sentencepiece.SpTextEmbedding

Get SentencePiece TextEmbeeding from SpTokenizer.

from(SpTokenizer) - Static method in class ai.djl.sentencepiece.SpVocabulary

Get Vocabulary from SpTokenizer.

G

getIndex(String) - Method in class ai.djl.sentencepiece.SpVocabulary

getProcessor() - Method in class ai.djl.sentencepiece.SpTokenizer

Get SentencePiece processor.

getToken(long) - Method in class ai.djl.sentencepiece.SpVocabulary

I

idToPiece(long, int) - Method in class ai.djl.sentencepiece.jni.SentencePieceLibrary

L

LIB - Static variable in class ai.djl.sentencepiece.jni.SentencePieceLibrary

LibUtils - Class in ai.djl.sentencepiece.jni

Utilities for finding the SentencePiece binary on the System.

loadLibrary() - Static method in class ai.djl.sentencepiece.jni.LibUtils

loadModel(long, String) - Method in class ai.djl.sentencepiece.jni.SentencePieceLibrary

loadModelFromBytes(long, byte[]) - Method in class ai.djl.sentencepiece.jni.SentencePieceLibrary

P

pieceToId(long, String) - Method in class ai.djl.sentencepiece.jni.SentencePieceLibrary

preprocessTextToEmbed(List<String>) - Method in class ai.djl.sentencepiece.SpTextEmbedding

S

SentencePieceLibrary - Class in ai.djl.sentencepiece.jni

A class containing utilities to interact with the SentencePiece Engine's JNI layer.

size() - Method in class ai.djl.sentencepiece.SpVocabulary

SpTextEmbedding - Class in ai.djl.sentencepiece

A TextEmbedding in SentencePiece that do sentence tokenization and map tokens into indices.

SpTokenizer - Class in ai.djl.sentencepiece

SpTokenizer is a SentencePiece implementation of the Tokenizer interface that converts sentences into token.

SpTokenizer(byte[]) - Constructor for class ai.djl.sentencepiece.SpTokenizer

Creates a SentencePiece Tokenizer from byte array.

SpTokenizer(InputStream) - Constructor for class ai.djl.sentencepiece.SpTokenizer

Creates a SentencePiece Tokenizer from inputStream.

SpTokenizer(Path) - Constructor for class ai.djl.sentencepiece.SpTokenizer

Create a SentencePiece Tokenizer from existing models.

SpTokenizer(Path, String) - Constructor for class ai.djl.sentencepiece.SpTokenizer

Create a SentencePiece Tokenizer from existing models.

SpVocabulary - Class in ai.djl.sentencepiece

SpVocabulary is a SentencePiece implementation of Vocabulary.

T

tokenize(long, String) - Method in class ai.djl.sentencepiece.jni.SentencePieceLibrary

tokenize(String) - Method in class ai.djl.sentencepiece.SpTokenizer

U

unembedText(NDArray) - Method in class ai.djl.sentencepiece.SpTextEmbedding

A B C D E F G I L P S T U
All Classes All Packages