dev.langchain4j.model.openai

Class OpenAiTokenizer

java.lang.Object

dev.langchain4j.model.openai.OpenAiTokenizer

All Implemented Interfaces:

dev.langchain4j.model.Tokenizer

public class OpenAiTokenizer
extends Object
implements dev.langchain4j.model.Tokenizer

This class can be used to estimate the cost (in tokens) before calling OpenAI or when using streaming. Magic numbers present in this class were found empirically while testing. There are integration tests in place that are making sure that the calculations here are very close to that of OpenAI.

Constructor Summary

Constructors

Constructor and Description
OpenAiTokenizer(String modelName)

Method Summary

Modifier and Type	Method and Description
String	decode(List<Integer> tokens)
List<Integer>	encode(String text)
List<Integer>	encode(String text, int maxTokensToEncode)
int	estimateTokenCountInForcefulToolExecutionRequest(dev.langchain4j.agent.tool.ToolExecutionRequest toolExecutionRequest)
int	estimateTokenCountInForcefulToolSpecification(dev.langchain4j.agent.tool.ToolSpecification toolSpecification)
int	estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
int	estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
int	estimateTokenCountInText(String text)
int	estimateTokenCountInToolExecutionRequests(Iterable<dev.langchain4j.agent.tool.ToolExecutionRequest> toolExecutionRequests)
int	estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface dev.langchain4j.model.Tokenizer

estimateTokenCountInTools, estimateTokenCountInTools

Constructor Detail

OpenAiTokenizer

public OpenAiTokenizer(String modelName)

Method Detail

estimateTokenCountInText

public int estimateTokenCountInText(String text)

Specified by:

estimateTokenCountInText in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessage

public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)

Specified by:

estimateTokenCountInMessage in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessages

public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)

Specified by:

estimateTokenCountInMessages in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInToolSpecifications

public int estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)

Specified by:

estimateTokenCountInToolSpecifications in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInForcefulToolSpecification

public int estimateTokenCountInForcefulToolSpecification(dev.langchain4j.agent.tool.ToolSpecification toolSpecification)

Specified by:

estimateTokenCountInForcefulToolSpecification in interface dev.langchain4j.model.Tokenizer

encode

public List<Integer> encode(String text)

encode

public List<Integer> encode(String text,
                            int maxTokensToEncode)

decode

public String decode(List<Integer> tokens)

estimateTokenCountInToolExecutionRequests

public int estimateTokenCountInToolExecutionRequests(Iterable<dev.langchain4j.agent.tool.ToolExecutionRequest> toolExecutionRequests)

Specified by:

estimateTokenCountInToolExecutionRequests in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInForcefulToolExecutionRequest

public int estimateTokenCountInForcefulToolExecutionRequest(dev.langchain4j.agent.tool.ToolExecutionRequest toolExecutionRequest)

Specified by:

estimateTokenCountInForcefulToolExecutionRequest in interface dev.langchain4j.model.Tokenizer