BertTokenizer (langchain4j-embeddings 0.19.0 API)

java.lang.Object

dev.langchain4j.model.embedding.BertTokenizer

All Implemented Interfaces:: dev.langchain4j.model.Tokenizer

public class BertTokenizer extends Object implements dev.langchain4j.model.Tokenizer

Constructor Summary

Constructors

Constructor

Description

BertTokenizer()
Method Summary

Modifier and Type

Method

Description

int

estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)

int

estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)

int

estimateTokenCountInText(String text)

int

estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)

long

tokenId(String token)

List<String>

tokenize(String text)

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface dev.langchain4j.model.Tokenizer
estimateTokenCountInTools, estimateTokenCountInTools, estimateTokenCountInToolSpecification

Constructor Details
- BertTokenizer
  
  public BertTokenizer()
Method Details
- estimateTokenCountInText
  
  public int estimateTokenCountInText(String text)
  
  Specified by:
  
  estimateTokenCountInText in interface dev.langchain4j.model.Tokenizer
- estimateTokenCountInMessage
  
  public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
  
  Specified by:
  
  estimateTokenCountInMessage in interface dev.langchain4j.model.Tokenizer
- estimateTokenCountInMessages
  
  public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
  
  Specified by:
  
  estimateTokenCountInMessages in interface dev.langchain4j.model.Tokenizer
- estimateTokenCountInToolSpecifications
  
  public int estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)
  
  Specified by:
  
  estimateTokenCountInToolSpecifications in interface dev.langchain4j.model.Tokenizer
- tokenize
  
  public List<String> tokenize(String text)
- tokenId
  
  public long tokenId(String token)