com.yahoo.prelude.query.parser

Class AbstractParser

java.lang.Object

com.yahoo.prelude.query.parser.AbstractParser

All Implemented Interfaces:

CustomParser, Parser

Direct Known Subclasses:

PhraseParser, StructuredParser

public abstract class AbstractParser
extends Object
implements CustomParser

The Vespa query parser.

Author:

bratseth, Steinar Knutsen

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) class	AbstractParser.Submodes An enumeration of the parser index-controlled submodes.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	braceLevelURL The counter for braces in URLs, braces in URLs are accepted so long as they are balanced.
protected ParserEnvironment	environment
protected IndexFacts.Session	indexFacts The IndexFacts.Session of this query
protected com.yahoo.language.Language	language The current language of this parser.
protected AbstractParser.Submodes	submodes The current submodes of this parser
protected TokenPosition	tokens

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractParser(ParserEnvironment environment) Creates a new instance of this class, storing the given ParserEnvironment for parse-time access to the environment.

Method Summary

Modifier and Type	Method and Description
private static void	assignDefaultIndex(String defaultIndex, Item item) Assigns the default index to query terms having no default index The parser _should_ have done this, for some reason it doesn't
private static Item	collapsePhrase(PhraseItem phrase)
private String	generateLanguageDetectionTextFrom(TokenPosition tokens, IndexFacts.Session indexFacts, String defaultIndex) Do a best-effort attempt at creating a single string for language detection from only the relevant subset of tokens.
private boolean	is(Token.Kind kind, Token tokenOrNull)
protected String	normalize(String input) Unicode normalizes some piece of natural language text.
QueryTree	parse(Parsable query) Parser the given Parsable, and returns a corresponding QueryTree.
Item	parse(String queryToParse, String filterToParse, com.yahoo.language.Language parsingLanguage, IndexFacts.Session indexFacts, String defaultIndexName)
private Item	parse(String queryToParse, String filterToParse, com.yahoo.language.Language parsingLanguage, IndexFacts.Session indexFacts, String defaultIndexName, Parsable parsable)
protected abstract Item	parseItems()
protected Item	segment(Token token)
protected void	setState(com.yahoo.language.Language queryLanguage, IndexFacts.Session indexFacts)
static Item	simplifyPhrases(Item unwashed) Collapses single item phrases in the tree to the contained item.
protected void	tokenize(String query, String defaultIndexName, IndexFacts.Session indexFacts, com.yahoo.language.Language language) Tokenizes the given string and initializes tokens with the found tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.yahoo.prelude.query.parser.CustomParser

parse

Field Detail

submodes

protected AbstractParser.Submodes submodes

The current submodes of this parser

language

protected com.yahoo.language.Language language

The current language of this parser. Used to decide whether and how to use the CJKSegmenter

indexFacts

protected IndexFacts.Session indexFacts

The IndexFacts.Session of this query

braceLevelURL

protected int braceLevelURL

The counter for braces in URLs, braces in URLs are accepted so long as they are balanced.

environment

protected final ParserEnvironment environment

tokens

protected final TokenPosition tokens

Constructor Detail

AbstractParser

protected AbstractParser(ParserEnvironment environment)

Creates a new instance of this class, storing the given ParserEnvironment for parse-time access to the environment.

Parameters:

environment - The environment settings to attach to the Parser.

Method Detail

parse

public final QueryTree parse(Parsable query)

Description copied from interface: Parser

Parser the given Parsable, and returns a corresponding QueryTree. If parsing fails without an exception, the contained root will be an instance of NullItem.

Specified by:

parse in interface Parser

Parameters:

query - the Parsable to parse

Returns:

the parsed QueryTree, never null

parse

public final Item parse(String queryToParse,
                        String filterToParse,
                        com.yahoo.language.Language parsingLanguage,
                        IndexFacts.Session indexFacts,
                        String defaultIndexName)

Specified by:

parse in interface CustomParser

parse

private Item parse(String queryToParse,
                   String filterToParse,
                   com.yahoo.language.Language parsingLanguage,
                   IndexFacts.Session indexFacts,
                   String defaultIndexName,
                   Parsable parsable)

generateLanguageDetectionTextFrom

private String generateLanguageDetectionTextFrom(TokenPosition tokens,
                                                 IndexFacts.Session indexFacts,
                                                 String defaultIndex)

Do a best-effort attempt at creating a single string for language detection from only the relevant subset of tokens. The relevant tokens are text tokens which follows names of indexes which are tokenized. This method does not modify the position of the given token stream.

is

private boolean is(Token.Kind kind,
                   Token tokenOrNull)

parseItems

protected abstract Item parseItems()

assignDefaultIndex

private static void assignDefaultIndex(String defaultIndex,
                                       Item item)

Assigns the default index to query terms having no default index The parser _should_ have done this, for some reason it doesn't

Parameters:

defaultIndex - The default index to assign.

item - The item to check.

normalize

protected String normalize(String input)

Unicode normalizes some piece of natural language text. The chosen form is compatibility decomposition, canonical composition (NFKC).

Parameters:

input - The string to normalize.

Returns:

The normalized string.

setState

protected void setState(com.yahoo.language.Language queryLanguage,
                        IndexFacts.Session indexFacts)

tokenize

protected void tokenize(String query,
                        String defaultIndexName,
                        IndexFacts.Session indexFacts,
                        com.yahoo.language.Language language)

Tokenizes the given string and initializes tokens with the found tokens.

Parameters:

query - the string to tokenize.

defaultIndexName - the name of the index to use as default.

indexFacts - resolved information about the index we are searching

language - the language set for this query, or null if none

simplifyPhrases

public static Item simplifyPhrases(Item unwashed)

Collapses single item phrases in the tree to the contained item.

Parameters:

unwashed - The item whose phrases to simplify.

Returns:

The simplified item.

collapsePhrase

private static Item collapsePhrase(PhraseItem phrase)

segment

protected Item segment(Token token)