Package dev.langchain4j.model.embedding.onnx

Class BertTokenizer

java.lang.Object
dev.langchain4j.model.embedding.onnx.BertTokenizer
All Implemented Interfaces:
dev.langchain4j.model.Tokenizer
@Deprecated(forRemoval=true) public class BertTokenizer extends Object implements dev.langchain4j.model.Tokenizer
Deprecated, for removal: This API element is subject to removal in a future version.
Use HuggingFaceTokenizer instead.

  • Constructor Summary

    Constructors
    Constructor
    Description
    BertTokenizer()
    Deprecated, for removal: This API element is subject to removal in a future version.
     
    BertTokenizer(URL vocabularyFile)
    Deprecated, for removal: This API element is subject to removal in a future version.
     

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
    Deprecated, for removal: This API element is subject to removal in a future version.
     
    int
    estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
    Deprecated, for removal: This API element is subject to removal in a future version.
     
    int
    estimateTokenCountInText(String text)
    Deprecated, for removal: This API element is subject to removal in a future version.
     
    long
    tokenId(String token)
    Deprecated, for removal: This API element is subject to removal in a future version.
     
    List<String>
    tokenize(String text)
    Deprecated, for removal: This API element is subject to removal in a future version.
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • BertTokenizer

      public BertTokenizer()
      Deprecated, for removal: This API element is subject to removal in a future version.

    • BertTokenizer

      public BertTokenizer(URL vocabularyFile)
      Deprecated, for removal: This API element is subject to removal in a future version.

  • Method Details

    • estimateTokenCountInText

      public int estimateTokenCountInText(String text)
      Deprecated, for removal: This API element is subject to removal in a future version.
      Specified by:
      estimateTokenCountInText in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInMessage

      public int estimateTokenCountInMessage(dev.langchain4j.data.message.ChatMessage message)
      Deprecated, for removal: This API element is subject to removal in a future version.
      Specified by:
      estimateTokenCountInMessage in interface dev.langchain4j.model.Tokenizer

    • estimateTokenCountInMessages

      public int estimateTokenCountInMessages(Iterable<dev.langchain4j.data.message.ChatMessage> messages)
      Deprecated, for removal: This API element is subject to removal in a future version.
      Specified by:
      estimateTokenCountInMessages in interface dev.langchain4j.model.Tokenizer

    • tokenize

      public List<String> tokenize(String text)
      Deprecated, for removal: This API element is subject to removal in a future version.

    • tokenId

      public long tokenId(String token)
      Deprecated, for removal: This API element is subject to removal in a future version.