Deep Java Library 0.35.0 - tokenizers API

This document is the API specification for the Deep Java Library (DJL) Huggingface tokenizers extension.

The Tokeniszers module contains the DJL extension for Huggingface tokenizers.

Packages

Package	Description
ai.djl.engine.rust	Contains classes to interface with the underlying native engine.
ai.djl.engine.rust.zoo	Contains the built-in RsModelZoo.
ai.djl.huggingface.tokenizers	Contains classes to interface with the underlying Huggingface tokenizers.
ai.djl.huggingface.tokenizers.jni	Contains classes to interface with the underlying Huggingface tokenizers.
ai.djl.huggingface.translator	Contains Translator classes that leverage Huggingface tokenizers.
ai.djl.huggingface.zoo	Contains the built-in HfModelZoo.