Package opennlp.tools.tokenize

Class TokenSample

java.lang.Object

opennlp.tools.tokenize.TokenSample

public class TokenSample
extends java.lang.Object

A TokenSample is text with token spans.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	DEFAULT_SEPARATOR_CHARS

Constructor Summary

Constructors

Constructor	Description
TokenSample(java.lang.String text, Span[] tokenSpans)	Initializes the current instance.
TokenSample(Detokenizer detokenizer, java.lang.String[] tokens)

Method Summary

Modifier and Type	Method	Description
boolean	equals(java.lang.Object obj)
java.lang.String	getText()	Retrieves the text.
Span[]	getTokenSpans()	Retrieves the token spans.
static TokenSample	parse(java.lang.String sampleString, java.lang.String separatorChars)
java.lang.String	toString()

Methods inherited from class java.lang.Object

getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_SEPARATOR_CHARS

public static final java.lang.String DEFAULT_SEPARATOR_CHARS

See Also:

Constant Field Values

Constructor Detail

TokenSample

public TokenSample(java.lang.String text,
                   Span[] tokenSpans)

Initializes the current instance.

Parameters:

text - the text which contains the tokens.

tokenSpans - the spans which mark the begin and end of the tokens.

TokenSample

public TokenSample(Detokenizer detokenizer,
                   java.lang.String[] tokens)

Method Detail

getText

public java.lang.String getText()

Retrieves the text.

getTokenSpans

public Span[] getTokenSpans()

Retrieves the token spans.

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

parse

public static TokenSample parse(java.lang.String sampleString,
                                java.lang.String separatorChars)

equals

public boolean equals(java.lang.Object obj)

Overrides:

equals in class java.lang.Object