dev.langchain4j.model.openai

Class OpenAiTokenizer

java.lang.Object

dev.langchain4j.model.openai.OpenAiTokenizer

All Implemented Interfaces:

dev.langchain4j.model.Tokenizer

public class OpenAiTokenizer
extends Object
implements dev.langchain4j.model.Tokenizer

Constructor Summary

Constructors

Constructor and Description
OpenAiTokenizer(String modelName)

Method Summary

Modifier and Type	Method and Description
String	decode(List<Integer> tokens)
List<Integer>	encode(String text)
List<Integer>	encode(String text, int maxTokensToEncode)
int	estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
int	estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
int	estimateTokenCountInText(String text)
int	estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface dev.langchain4j.model.Tokenizer

estimateTokenCountInTools, estimateTokenCountInTools, estimateTokenCountInToolSpecification

Constructor Detail

OpenAiTokenizer

public OpenAiTokenizer(String modelName)

Method Detail

estimateTokenCountInText

public int estimateTokenCountInText(String text)

Specified by:

estimateTokenCountInText in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessage

public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)

Specified by:

estimateTokenCountInMessage in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessages

public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)

Specified by:

estimateTokenCountInMessages in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInToolSpecifications

public int estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)

Specified by:

estimateTokenCountInToolSpecifications in interface dev.langchain4j.model.Tokenizer

encode

public List<Integer> encode(String text)

encode

public List<Integer> encode(String text,
                            int maxTokensToEncode)

decode

public String decode(List<Integer> tokens)