Package dev.langchain4j.model.embedding.onnx

Class HuggingFaceTokenizer

java.lang.Object

dev.langchain4j.model.embedding.onnx.HuggingFaceTokenizer

All Implemented Interfaces:

dev.langchain4j.model.Tokenizer

public class HuggingFaceTokenizer
extends Object
implements dev.langchain4j.model.Tokenizer

A HuggingFace tokenizer.
Uses DJL's HuggingFaceTokenizer under the hood.
Requires tokenizer.json to instantiate. An example.

Constructor Summary

Constructors

Constructor	Description
HuggingFaceTokenizer()	Creates an instance of a HuggingFaceTokenizer using a built-in tokenizer.json file.
HuggingFaceTokenizer(String pathToTokenizer)	Creates an instance of a HuggingFaceTokenizer using a provided tokenizer.json file.
HuggingFaceTokenizer(String pathToTokenizer, Map<String,String> options)	Creates an instance of a HuggingFaceTokenizer using a provided tokenizer.json file and a map of DJL's tokenizer options.
HuggingFaceTokenizer(Path pathToTokenizer)	Creates an instance of a HuggingFaceTokenizer using a provided tokenizer.json file.
HuggingFaceTokenizer(Path pathToTokenizer, Map<String,String> options)	Creates an instance of a HuggingFaceTokenizer using a provided tokenizer.json file and a map of DJL's tokenizer options.

Method Summary

Modifier and Type	Method	Description
int	estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
int	estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
int	estimateTokenCountInText(String text)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

HuggingFaceTokenizer

public HuggingFaceTokenizer()

Creates an instance of a HuggingFaceTokenizer using a built-in tokenizer.json file.

HuggingFaceTokenizer

public HuggingFaceTokenizer(Path pathToTokenizer)

Creates an instance of a HuggingFaceTokenizer using a provided tokenizer.json file.

Parameters:

pathToTokenizer - The path to the tokenizer file (e.g., "/path/to/tokenizer.json")

HuggingFaceTokenizer

public HuggingFaceTokenizer(Path pathToTokenizer,
 Map<String,String> options)

Creates an instance of a HuggingFaceTokenizer using a provided tokenizer.json file and a map of DJL's tokenizer options.

Parameters:

pathToTokenizer - The path to the tokenizer file (e.g., "/path/to/tokenizer.json")

options - The DJL's tokenizer options

HuggingFaceTokenizer

public HuggingFaceTokenizer(String pathToTokenizer)

Creates an instance of a HuggingFaceTokenizer using a provided tokenizer.json file.

Parameters:

pathToTokenizer - The path to the tokenizer file (e.g., "/path/to/tokenizer.json")

HuggingFaceTokenizer

public HuggingFaceTokenizer(String pathToTokenizer,
 Map<String,String> options)

Creates an instance of a HuggingFaceTokenizer using a provided tokenizer.json file and a map of DJL's tokenizer options.

Parameters:

pathToTokenizer - The path to the tokenizer file (e.g., "/path/to/tokenizer.json")

options - The DJL's tokenizer options

Method Details

estimateTokenCountInText

public int estimateTokenCountInText(String text)

Specified by:

estimateTokenCountInText in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessage

public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)

Specified by:

estimateTokenCountInMessage in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessages

public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)

Specified by:

estimateTokenCountInMessages in interface dev.langchain4j.model.Tokenizer