preprocess

Type Members

1. class JavaSentenceSegmenter extends SentenceSegmenter

   A Sentence Segmenter backed by Java's BreakIterator.

2. class JavaWordTokenizer extends Tokenizer

   A Word Segmenter backed by Java's BreakIterator.

3. class MLSentenceSegmenter extends SentenceSegmenter with Serializable

   Annotations

   @SerialVersionUID( 1L )

4. class NewLineSentenceSegmenter extends SentenceSegmenter

   TODO move to chalk

5. case class RegexSearchTokenizer(pattern: String) extends Tokenizer with Product with Serializable

   Finds all occurrences of the given pattern in the document.

6. case class RegexSplitTokenizer(pattern: String) extends Tokenizer with Product with Serializable

   Splits the input document according to the given pattern.

7. class SegmentingIterator extends Iterator[Span]

8. trait SentenceSegmenter extends StringAnalysisFunction[Any, Sentence] with (String) ⇒ Iterable[String] with Serializable

9. class StreamSentenceSegmenter extends AnyRef

   TODO

10. trait Tokenizer extends StringAnalysisFunction[Sentence, Token] with Serializable with (String) ⇒ IndexedSeq[String]

    Abstract trait for tokenizers, which annotate sentence-segmented text with tokens.

11. class TreebankTokenizer extends Tokenizer with Serializable

    Annotations

    @SerialVersionUID( 1L )

12. class WhitespaceTokenizer extends RegexSplitTokenizer

    Tokenizes by splitting on the regular expression \s+.