Tokenizer

All Implemented Interfaces:
```
public interface Tokenizer

                    
```
Represents an interface for estimating the count of tokens in various text types such as a text, prompt, text segment, etc. This can be useful when it's necessary to know in advance the cost of processing a specified text by the LLM.

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description

Field Summary

Fields
Modifier and Type	Field	Description

Constructor Summary

Constructors
Constructor	Description

Enum Constant Summary

Enum Constants
Enum Constant	Description

Method Summary

Modifier and Type	Method	Description
`abstract int`	`estimateTokenCountInText(String text)`	Estimates the count of tokens in the given text.
`abstract int`	`estimateTokenCountInMessage(ChatMessage message)`	Estimates the count of tokens in the given message.
`abstract int`	`estimateTokenCountInMessages(Iterable<ChatMessage> messages)`	Estimates the count of tokens in the given messages.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
- Method Detail
  - estimateTokenCountInText
```
 abstract int estimateTokenCountInText(String text)
```
    Estimates the count of tokens in the given text.
    
    Parameters:
    
    text - the text.
    
    Returns:
    
    the estimated count of tokens.
  - estimateTokenCountInMessage
```
 abstract int estimateTokenCountInMessage(ChatMessage message)
```
    Estimates the count of tokens in the given message.
    
    Parameters:
    
    message - the message.
    
    Returns:
    
    the estimated count of tokens.
  - estimateTokenCountInMessages
```
 abstract int estimateTokenCountInMessages(Iterable<ChatMessage> messages)
```
    Estimates the count of tokens in the given messages.
    
    Parameters:
    
    messages - the messages.
    
    Returns:
    
    the estimated count of tokens.