This document is the API specification for the Deep Java Library (DJL) Huggingface tokenizers extension.
The Tokeniszers module contains the DJL extension for Huggingface tokenizers.
Package | Description |
---|---|
ai.djl.huggingface.tokenizers |
Contains classes to interface with the underlying Huggingface tokenizers.
|
ai.djl.huggingface.tokenizers.jni |
Contains classes to interface with the underlying Huggingface tokenizers.
|
ai.djl.huggingface.translator |
Contains
Translator classes that leverage Huggingface tokenizers. |
ai.djl.huggingface.zoo |
Contains the built-in
HfModelZoo . |