Package co.elastic.clients.elasticsearch.ml

Class CommonTokenizationConfig

java.lang.Object

co.elastic.clients.elasticsearch.ml.CommonTokenizationConfig

All Implemented Interfaces:

JsonpSerializable

Direct Known Subclasses:

NlpBertTokenizationConfig, NlpRobertaTokenizationConfig, XlmRobertaTokenizationConfig

public abstract class CommonTokenizationConfig
extends Object
implements JsonpSerializable

See Also:

• API specification

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	CommonTokenizationConfig.AbstractBuilder<BuilderT extends CommonTokenizationConfig.AbstractBuilder<BuilderT>>

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	CommonTokenizationConfig(CommonTokenizationConfig.AbstractBuilder<?> builder)

Method Summary

Modifier and Type	Method	Description
final Boolean	doLowerCase()	Should the tokenizer lower case the text
final Integer	maxSequenceLength()	Maximum input sequence length for the model
void	serialize(jakarta.json.stream.JsonGenerator generator, JsonpMapper mapper)	Serialize this object to JSON.
protected void	serializeInternal(jakarta.json.stream.JsonGenerator generator, JsonpMapper mapper)
protected static <BuilderT extends CommonTokenizationConfig.AbstractBuilder<BuilderT>> void	setupCommonTokenizationConfigDeserializer(ObjectDeserializer<BuilderT> op)
final Integer	span()	Tokenization spanning options.
String	toString()
final TokenizationTruncate	truncate()	Should tokenization input be automatically truncated before sending to the model for inference
final Boolean	withSpecialTokens()	Is tokenization completed with special tokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

CommonTokenizationConfig

protected CommonTokenizationConfig(CommonTokenizationConfig.AbstractBuilder<?> builder)

Method Details

doLowerCase

@Nullable
public final Boolean doLowerCase()

Should the tokenizer lower case the text

API name: do_lower_case

maxSequenceLength

@Nullable
public final Integer maxSequenceLength()

Maximum input sequence length for the model

API name: max_sequence_length

span

@Nullable
public final Integer span()

Tokenization spanning options. Special value of -1 indicates no spanning takes place

API name: span

truncate

@Nullable
public final TokenizationTruncate truncate()

Should tokenization input be automatically truncated before sending to the model for inference

API name: truncate

withSpecialTokens

@Nullable
public final Boolean withSpecialTokens()

Is tokenization completed with special tokens

API name: with_special_tokens

serialize

public void serialize(jakarta.json.stream.JsonGenerator generator,
 JsonpMapper mapper)

Serialize this object to JSON.

Specified by:

serialize in interface JsonpSerializable

serializeInternal

protected void serializeInternal(jakarta.json.stream.JsonGenerator generator,
 JsonpMapper mapper)

toString

public String toString()

Overrides:

toString in class Object

setupCommonTokenizationConfigDeserializer

protected static <BuilderT extends CommonTokenizationConfig.AbstractBuilder<BuilderT>>
void setupCommonTokenizationConfigDeserializer(ObjectDeserializer<BuilderT> op)