Package dev.langchain4j.model.openai

Class OpenAiTokenizer

java.lang.Object
dev.langchain4j.model.openai.OpenAiTokenizer
All Implemented Interfaces:
dev.langchain4j.model.Tokenizer
public class OpenAiTokenizer extends Object implements dev.langchain4j.model.Tokenizer
This class can be used to estimate the cost (in tokens) before calling OpenAI or when using streaming. Magic numbers present in this class were found empirically while testing. There are integration tests in place that are making sure that the calculations here are very close to that of OpenAI.

  • Constructor Details

    • OpenAiTokenizer

      public OpenAiTokenizer()
      Creates an instance of the OpenAiTokenizer for the "gpt-3.5-turbo" model. It should be suitable for all current OpenAI models, as they all use the same cl100k_base encoding.

    • OpenAiTokenizer

      public OpenAiTokenizer(OpenAiChatModelName modelName)
      Creates an instance of the OpenAiTokenizer for a given OpenAiChatModelName.

    • OpenAiTokenizer

      public OpenAiTokenizer(OpenAiEmbeddingModelName modelName)
      Creates an instance of the OpenAiTokenizer for a given OpenAiEmbeddingModelName.

    • OpenAiTokenizer

      public OpenAiTokenizer(OpenAiLanguageModelName modelName)
      Creates an instance of the OpenAiTokenizer for a given OpenAiLanguageModelName.

    • OpenAiTokenizer

      public OpenAiTokenizer(String modelName)
      Creates an instance of the OpenAiTokenizer for a given model name.

  • Method Details

    • estimateTokenCountInText

      public int estimateTokenCountInText(String text)
      Specified by:
      estimateTokenCountInText in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInMessage

      public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
      Specified by:
      estimateTokenCountInMessage in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInMessages

      public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
      Specified by:
      estimateTokenCountInMessages in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInToolSpecifications

      public int estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)
      Specified by:
      estimateTokenCountInToolSpecifications in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInForcefulToolSpecification

      public int estimateTokenCountInForcefulToolSpecification(dev.langchain4j.agent.tool.ToolSpecification toolSpecification)
      Specified by:
      estimateTokenCountInForcefulToolSpecification in interface dev.langchain4j.model.Tokenizer

    • encode

      public List<Integer> encode(String text)

    • encode

      public List<Integer> encode(String text, int maxTokensToEncode)

    • decode

      public String decode(List<Integer> tokens)

    • estimateTokenCountInToolExecutionRequests

      public int estimateTokenCountInToolExecutionRequests(Iterable<dev.langchain4j.agent.tool.ToolExecutionRequest> toolExecutionRequests)
      Specified by:
      estimateTokenCountInToolExecutionRequests in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInForcefulToolExecutionRequest

      public int estimateTokenCountInForcefulToolExecutionRequest(dev.langchain4j.agent.tool.ToolExecutionRequest toolExecutionRequest)
      Specified by:
      estimateTokenCountInForcefulToolExecutionRequest in interface dev.langchain4j.model.Tokenizer