A B C D E G H L N T
All Classes All Packages

A

ai.djl.huggingface.tokenizers - package ai.djl.huggingface.tokenizers

Contains classes to interface with the underlying Huggingface tokenizers.

ai.djl.huggingface.tokenizers.jni - package ai.djl.huggingface.tokenizers.jni

Contains classes to interface with the underlying Huggingface tokenizers.

B

batchEncode(long, String[], boolean) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

batchEncode(String[]) - Method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

Returns the Encoding of the input sentence in batch.

batchEncode(List<String>) - Method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

Returns the Encoding of the input sentence in batch.

buildSentence(List<String>) - Method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

C

checkStatus() - Static method in class ai.djl.huggingface.tokenizers.jni.LibUtils

close() - Method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

createTokenizer(String) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

createTokenizerFromString(String) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

D

deleteEncoding(long) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

deleteTokenizer(long) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

E

encode(long, String, boolean) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

encode(String) - Method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

Returns the Encoding of the input sentence.

encode(String[]) - Method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

Returns the Encoding of the input sentences.

encode(List<String>) - Method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

Returns the Encoding of the input sentences.

encodeList(long, String[], boolean) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

Encoding - Class in ai.djl.huggingface.tokenizers

A class holds token encoding information.

G

getAttentionMask() - Method in class ai.djl.huggingface.tokenizers.Encoding

Returns the attention masks.

getAttentionMask(long) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

getIds() - Method in class ai.djl.huggingface.tokenizers.Encoding

Returns the token ids.

getSpecialTokenMask() - Method in class ai.djl.huggingface.tokenizers.Encoding

Returns the special token masks.

getSpecialTokenMask(long) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

getTokenIds(long) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

getTokens() - Method in class ai.djl.huggingface.tokenizers.Encoding

Returns the tokens.

getTokens(long) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

getTypeIds() - Method in class ai.djl.huggingface.tokenizers.Encoding

Returns the token type ids.

getTypeIds(long) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

getWordIds() - Method in class ai.djl.huggingface.tokenizers.Encoding

Returns the word ids.

getWordIds(long) - Method in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

H

HuggingFaceTokenizer - Class in ai.djl.huggingface.tokenizers

HuggingFaceTokenizer is a Huggingface tokenizer implementation of the Tokenizer interface that converts sentences into token.

L

LIB - Static variable in class ai.djl.huggingface.tokenizers.jni.TokenizersLibrary

LibUtils - Class in ai.djl.huggingface.tokenizers.jni

Utilities for finding the Huggingface tokenizer native binary on the System.

N

newInstance(InputStream, Map<String, String>) - Static method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

Create a pre-trained HuggingFaceTokenizer instance from InputStream.

newInstance(String) - Static method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

Create a pre-trained HuggingFaceTokenizer instance from hugginface hub.

newInstance(String, Map<String, String>) - Static method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

Create a pre-trained HuggingFaceTokenizer instance from hugginface hub.

newInstance(Path) - Static method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

Create a pre-trained HuggingFaceTokenizer instance from existing models.

newInstance(Path, Map<String, String>) - Static method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

Create a pre-trained HuggingFaceTokenizer instance from existing models.

T

tokenize(String) - Method in class ai.djl.huggingface.tokenizers.HuggingFaceTokenizer

TokenizersLibrary - Class in ai.djl.huggingface.tokenizers.jni

A class containing utilities to interact with the Tokenizer JNI layer.

A B C D E G H L N T
All Classes All Packages