Package dev.langchain4j.model.azure

Class AzureOpenAiTokenizer

java.lang.Object

dev.langchain4j.model.azure.AzureOpenAiTokenizer

All Implemented Interfaces:

dev.langchain4j.model.Tokenizer

public class AzureOpenAiTokenizer
extends Object
implements dev.langchain4j.model.Tokenizer

This class can be used to estimate the cost (in tokens) before calling OpenAI or when using streaming. Magic numbers present in this class were found empirically while testing. There are integration tests in place that are making sure that the calculations here are very close to that of OpenAI.

Constructor Summary

Constructors

Constructor	Description
AzureOpenAiTokenizer()	Creates an instance of the AzureOpenAiTokenizer for the "gpt-3.5-turbo" model.
AzureOpenAiTokenizer(AzureOpenAiChatModelName modelName)	Creates an instance of the AzureOpenAiTokenizer for a given AzureOpenAiChatModelName.
AzureOpenAiTokenizer(AzureOpenAiEmbeddingModelName modelName)	Creates an instance of the AzureOpenAiTokenizer for a given AzureOpenAiEmbeddingModelName.
AzureOpenAiTokenizer(AzureOpenAiLanguageModelName modelName)	Creates an instance of the AzureOpenAiTokenizer for a given AzureOpenAiLanguageModelName.
AzureOpenAiTokenizer(String modelName)	Creates an instance of the AzureOpenAiTokenizer for a given model name.

Method Summary

Modifier and Type	Method	Description
int	estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
int	estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
int	estimateTokenCountInText(String text)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

AzureOpenAiTokenizer

public AzureOpenAiTokenizer()

Creates an instance of the AzureOpenAiTokenizer for the "gpt-3.5-turbo" model. It should be suitable for most OpenAI models, as most of them use the same cl100k_base encoding (except for GPT-4o).

AzureOpenAiTokenizer

public AzureOpenAiTokenizer(AzureOpenAiChatModelName modelName)

Creates an instance of the AzureOpenAiTokenizer for a given AzureOpenAiChatModelName.

AzureOpenAiTokenizer

public AzureOpenAiTokenizer(AzureOpenAiEmbeddingModelName modelName)

Creates an instance of the AzureOpenAiTokenizer for a given AzureOpenAiEmbeddingModelName.

AzureOpenAiTokenizer

public AzureOpenAiTokenizer(AzureOpenAiLanguageModelName modelName)

Creates an instance of the AzureOpenAiTokenizer for a given AzureOpenAiLanguageModelName.

AzureOpenAiTokenizer

public AzureOpenAiTokenizer(String modelName)

Creates an instance of the AzureOpenAiTokenizer for a given model name.

Method Details

estimateTokenCountInText

public int estimateTokenCountInText(String text)

Specified by:

estimateTokenCountInText in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessage

public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)

Specified by:

estimateTokenCountInMessage in interface dev.langchain4j.model.Tokenizer

estimateTokenCountInMessages

public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)

Specified by:

estimateTokenCountInMessages in interface dev.langchain4j.model.Tokenizer