Package com.yahoo.language.simple

Class SimpleToken

java.lang.Object
com.yahoo.language.simple.SimpleToken
All Implemented Interfaces:
Token
public class SimpleToken extends Object implements Token
Author:
Mathias Mølster Lidal

  • Constructor Details

    • SimpleToken

      public SimpleToken(String original)

    • SimpleToken

      public SimpleToken(String original, String tokenString)

  • Method Details

    • getOrig

      public String getOrig()
      Description copied from interface: Token
      Returns the original form of this token
      Specified by:
      getOrig in interface Token

    • getNumStems

      public int getNumStems()
      Description copied from interface: Token
      Returns the number of stem forms available for this token.
      Specified by:
      getNumStems in interface Token

    • getStem

      public String getStem(int i)
      Description copied from interface: Token
      Returns the stem at position i
      Specified by:
      getStem in interface Token

    • getNumComponents

      public int getNumComponents()
      Description copied from interface: Token
      Returns the number of components, if this token is a compound word (e.g. german "kommunikationsfehler". Otherwise, return 0
      Specified by:
      getNumComponents in interface Token
      Returns:
      number of components, or 0 if none

    • getComponent

      public Token getComponent(int i)
      Description copied from interface: Token
      Returns a component token of this
      Specified by:
      getComponent in interface Token

    • addComponent

      public SimpleToken addComponent(Token token)

    • getTokenString

      public String getTokenString()
      Description copied from interface: Token
      Returns the token string in a form suitable for indexing: The most lowercased variant of the most processed token form available, If called on a compound token this returns a lowercased form of the entire word. If this is a special token with a configured replacement, this will return the replacement token.
      Specified by:
      getTokenString in interface Token

    • setTokenString

      public SimpleToken setTokenString(String str)

    • getType

      public TokenType getType()
      Description copied from interface: Token
      Returns the type of this token - word, space or punctuation etc.
      Specified by:
      getType in interface Token

    • setType

      public SimpleToken setType(TokenType type)

    • getScript

      public TokenScript getScript()
      Description copied from interface: Token
      Returns the script of this token
      Specified by:
      getScript in interface Token

    • setScript

      public SimpleToken setScript(TokenScript script)

    • isSpecialToken

      public boolean isSpecialToken()
      Description copied from interface: Token
      Returns whether this is an instance of a declared special token (e.g. c++)
      Specified by:
      isSpecialToken in interface Token

    • setSpecialToken

      public SimpleToken setSpecialToken(boolean specialToken)

    • getOffset

      public long getOffset()
      Description copied from interface: Token
      Returns the offset position of this token
      Specified by:
      getOffset in interface Token

    • setOffset

      public SimpleToken setOffset(long offset)

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • toDetailString

      public String toDetailString()

    • isIndexable

      public boolean isIndexable()
      Description copied from interface: Token
      Whether this token should be indexed
      Specified by:
      isIndexable in interface Token

    • fromStems

      public static SimpleToken fromStems(String original, List<String> stems)