dev.langchain4j.model

Interface Tokenizer

public interface Tokenizer

Represents an interface for estimating the count of tokens in various text types such as a text, prompt, text segment, etc. This can be useful when it's necessary to know in advance the cost of processing a specified text by the LLM.

Method Summary

Modifier and Type	Method and Description
default int	estimateTokenCountInForcefulToolExecutionRequest(ToolExecutionRequest toolExecutionRequest) Estimates the count of tokens in the given tool execution request.
default int	estimateTokenCountInForcefulToolSpecification(ToolSpecification toolSpecification) Estimates the count of tokens in the given tool specification.
int	estimateTokenCountInMessage(ChatMessage message) Estimates the count of tokens in the given message.
int	estimateTokenCountInMessages(Iterable<ChatMessage> messages) Estimates the count of tokens in the given messages.
int	estimateTokenCountInText(String text) Estimates the count of tokens in the given text.
int	estimateTokenCountInToolExecutionRequests(Iterable<ToolExecutionRequest> toolExecutionRequests) Estimates the count of tokens in the given tool execution requests.
default int	estimateTokenCountInTools(Iterable<Object> objectsWithTools) Estimates the count of tokens in Tool annotations of the given objects.
default int	estimateTokenCountInTools(Object objectWithTools) Estimates the count of tokens in Tool annotations of the given object.
int	estimateTokenCountInToolSpecifications(Iterable<ToolSpecification> toolSpecifications) Estimates the count of tokens in the given tool specifications.

Method Detail

estimateTokenCountInText

int estimateTokenCountInText(String text)

Estimates the count of tokens in the given text.

Parameters:

text - the text.

Returns:

the estimated count of tokens.

estimateTokenCountInMessage

int estimateTokenCountInMessage(ChatMessage message)

Estimates the count of tokens in the given message.

Parameters:

message - the message.

Returns:

the estimated count of tokens.

estimateTokenCountInMessages

int estimateTokenCountInMessages(Iterable<ChatMessage> messages)

Estimates the count of tokens in the given messages.

Parameters:

messages - the messages.

Returns:

the estimated count of tokens.

estimateTokenCountInTools

default int estimateTokenCountInTools(Object objectWithTools)

Estimates the count of tokens in Tool annotations of the given object.

Parameters:

objectWithTools - the object.

Returns:

the estimated count of tokens.

estimateTokenCountInTools

default int estimateTokenCountInTools(Iterable<Object> objectsWithTools)

Estimates the count of tokens in Tool annotations of the given objects.

Parameters:

objectsWithTools - the objects.

Returns:

the estimated count of tokens.

estimateTokenCountInToolSpecifications

int estimateTokenCountInToolSpecifications(Iterable<ToolSpecification> toolSpecifications)

Estimates the count of tokens in the given tool specifications.

Parameters:

toolSpecifications - the tool specifications.

Returns:

the estimated count of tokens.

estimateTokenCountInForcefulToolSpecification

default int estimateTokenCountInForcefulToolSpecification(ToolSpecification toolSpecification)

Estimates the count of tokens in the given tool specification.

Parameters:

toolSpecification - the tool specification.

Returns:

the estimated count of tokens.

estimateTokenCountInToolExecutionRequests

int estimateTokenCountInToolExecutionRequests(Iterable<ToolExecutionRequest> toolExecutionRequests)

Estimates the count of tokens in the given tool execution requests.

Parameters:

toolExecutionRequests - the tool execution request.

Returns:

the estimated count of tokens.

estimateTokenCountInForcefulToolExecutionRequest

default int estimateTokenCountInForcefulToolExecutionRequest(ToolExecutionRequest toolExecutionRequest)

Estimates the count of tokens in the given tool execution request.

Parameters:

toolExecutionRequest - the tool execution request.

Returns:

the estimated count of tokens.