Package org.basex.util.ft

Class WesternTokenizer

  • All Implemented Interfaces:
    Comparable<org.basex.util.ft.LanguageImpl>, Iterator<FTSpan>
    public final class WesternTokenizer
extends Tokenizer
    Full-text tokenizer.
    Author:
    BaseX Team 2005-24, BSD License, Christian Gruen

    • Constructor Detail

      • WesternTokenizer

        public WesternTokenizer​(FTOpt fto)
        Constructor.
        Parameters:
        fto - full-text options

    • Method Detail

      • init

        public WesternTokenizer init​(byte[] txt)
        Description copied from class: FTIterator
        Initializes the iterator.
        Specified by:
        init in class FTIterator
        Parameters:
        txt - text
        Returns:
        self reference

      • hasNext

        public boolean hasNext()

      • nextToken

        public byte[] nextToken()
        Description copied from class: FTIterator
        Returns the next token. May be called as an alternative to Iterator.next() to avoid the creation of new FTSpan instances.
        Specified by:
        nextToken in class FTIterator
        Returns:
        token

      • prec

        protected byte prec()
        Returns the precedence of the processor. If two language implementations support the same language, the processor with the higher precedence will be selected.
        Returns:
        precedence

      • supports

        public boolean supports​(Language ln)
        Checks if the specified language is supported.
        Parameters:
        ln - language
        Returns:
        true if language is supported

      • equals

        public final boolean equals​(Object obj)
        Overrides:
        equals in class Object

      • compareTo

        public final int compareTo​(org.basex.util.ft.LanguageImpl o)
        Specified by:
        compareTo in interface Comparable<org.basex.util.ft.LanguageImpl>

      • hashCode

        public int hashCode()
        Overrides:
        hashCode in class Object