Package dev.langchain4j.model.embedding.onnx

Class BertTokenizer

java.lang.Object
dev.langchain4j.model.embedding.onnx.BertTokenizer
All Implemented Interfaces:
dev.langchain4j.model.Tokenizer
@Deprecated public class BertTokenizer extends Object implements dev.langchain4j.model.Tokenizer
Deprecated.
Use HuggingFaceTokenizer instead.

  • Constructor Details

    • BertTokenizer

      public BertTokenizer()
      Deprecated.

    • BertTokenizer

      public BertTokenizer(URL vocabularyFile)
      Deprecated.

  • Method Details

    • estimateTokenCountInText

      public int estimateTokenCountInText(String text)
      Deprecated.
      Specified by:
      estimateTokenCountInText in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInMessage

      public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
      Deprecated.
      Specified by:
      estimateTokenCountInMessage in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInMessages

      public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
      Deprecated.
      Specified by:
      estimateTokenCountInMessages in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInToolSpecifications

      public int estimateTokenCountInToolSpecifications(Iterable<dev.langchain4j.agent.tool.ToolSpecification> toolSpecifications)
      Deprecated.
      Specified by:
      estimateTokenCountInToolSpecifications in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInToolExecutionRequests

      public int estimateTokenCountInToolExecutionRequests(Iterable<dev.langchain4j.agent.tool.ToolExecutionRequest> toolExecutionRequests)
      Deprecated.
      Specified by:
      estimateTokenCountInToolExecutionRequests in interface dev.langchain4j.model.Tokenizer

    • tokenize

      public List<String> tokenize(String text)
      Deprecated.

    • tokenId

      public long tokenId(String token)
      Deprecated.