This document is the API specification for the Deep Java Library (DJL) Huggingface tokenizers extension.

The Tokeniszers module contains the DJL extension for Huggingface tokenizers.

Packages

Package	Description
ai.djl.huggingface.tokenizers	Contains classes to interface with the underlying Huggingface tokenizers.
ai.djl.huggingface.tokenizers.jni	Contains classes to interface with the underlying Huggingface tokenizers.