Package org.apache.lucene.search.highlight
Example Usage
//... Above, create documents with two fields, one with term vectors (tv) and one without (notv) IndexSearcher searcher = new IndexSearcher(directory); QueryParser parser = new QueryParser("notv", analyzer); Query query = parser.parse("million"); TopDocs hits = searcher.search(query, 10); SimpleHTMLFormatter htmlFormatter = new SimpleHTMLFormatter(); Highlighter highlighter = new Highlighter(htmlFormatter, new QueryScorer(query)); for (int i = 0; i < 10; i++) { int id = hits.scoreDocs[i].doc; Document doc = searcher.doc(id); String text = doc.get("notv"); TokenStream tokenStream = TokenSources.getAnyTokenStream(searcher.getIndexReader(), id, "notv", analyzer); TextFragment[] frag = highlighter.getBestTextFragments(tokenStream, text, false, 10);//highlighter.getBestFragments(tokenStream, text, 3, "..."); for (int j = 0; j < frag.length; j++) { if ((frag[j] != null) && (frag[j].getScore() > 0)) { System.out.println((frag[j].toString())); } } //Term vector text = doc.get("tv"); tokenStream = TokenSources.getAnyTokenStream(searcher.getIndexReader(), hits.scoreDocs[i].doc, "tv", analyzer); frag = highlighter.getBestTextFragments(tokenStream, text, false, 10); for (int j = 0; j < frag.length; j++) { if ((frag[j] != null) && (frag[j].getScore() > 0)) { System.out.println((frag[j].toString())); } } System.out.println("-------------"); }
New features 06/02/2005
This release adds options for encoding (thanks to Nicko Cadell). An "Encoder" implementation such as the new SimpleHTMLEncoder class can be passed to the highlighter to encode all those non-xhtml standard characters such as & into legal values. This simple class may not suffice for some languages - Commons Lang has an implementation that could be used: escapeHtml(String) in http://svn.apache.org/viewcvs.cgi/jakarta/commons/proper/lang/trunk/src/java/org/apache/commons/lang/StringEscapeUtils.java?rev=137958&view=markupNew features 22/12/2004
This release adds some new capabilities:- Faster highlighting using Term vector support
- New formatting options to use color intensity to show informational value
- Options for better summarization by using term IDF scores to influence fragment selection
The highlighter takes a TokenStream as input. Until now these streams have typically been produced using an Analyzer but the new class TokenSources provides helper methods for obtaining TokenStreams from the new TermVector position support (see latest CVS version).
The new class GradientFormatter can use a scale of colors to highlight terms according to their score. A subtle use of color can help emphasise the reasons for matching (useful when doing "MoreLikeThis" queries and you want to see what the basis of the similarities are).
The QueryScorer class has a new constructor which can use an IndexReader to derive the IDF (inverse document frequency) for each term in order to influence the score. This is useful for helping to extracting the most significant sections of a document and in supplying scores used by the new GradientFormatter to color significant words more strongly. The QueryScorer.getMaxWeight method is useful when passed to the GradientFormatter constructor to define the top score which is associated with the top color.
-
Interface Summary Interface Description Encoder Encodes original text.Formatter Processes terms found in the original text, typically by applying some form of mark-up to highlight terms in HTML search results pages.Fragmenter Implements the policy for breaking text into multiple fragments for consideration by theHighlighter
class.Scorer A Scorer is responsible for scoring a stream of tokens. -
Class Summary Class Description DefaultEncoder SimpleEncoder
implementation that does not modify the outputGradientFormatter Formats text with different color intensity depending on the score of the term.Highlighter Class used to markup highlighted terms found in the best sections of a text, using configurableFragmenter
,Scorer
,Formatter
,Encoder
and tokenizers.NullFragmenter Fragmenter
implementation which does not fragment the text.OffsetLimitTokenFilter This TokenFilter limits the number of tokens while indexing by adding up the current offset.PositionSpan Utility class to record Positions SpansQueryScorer Scorer
implementation which scores text fragments by the number of unique query terms found.QueryTermExtractor Utility class used to extract the terms used in a query, plus any weights.QueryTermScorer Scorer
implementation which scores text fragments by the number of unique query terms found.SimpleFragmenter Fragmenter
implementation which breaks text up into same-size fragments with no concerns over spotting sentence boundaries.SimpleHTMLEncoder SimpleEncoder
implementation to escape text for HTML outputSimpleHTMLFormatter SimpleFormatter
implementation to highlight terms with a pre and post tag.SimpleSpanFragmenter Fragmenter
implementation which breaks text up into same-size fragments but does not split upSpans
.SpanGradientFormatter Formats text with different color intensity depending on the score of the term using the span tag.TextFragment Low-level class used to record information about a section of a document with a score.TokenGroup One, or several overlapping tokens, along with the score(s) and the scope of the original textTokenSources Hides implementation issues associated with obtaining a TokenStream for use with the higlighter - can obtain from TermFreqVectors with offsets and (optionally) positions or from Analyzer class reparsing the stored content.TokenStreamFromTermPositionVector TokenStream created from a term vector field.WeightedSpanTerm Lightweight class to hold term, weight, and positions used for scoring this term.WeightedSpanTermExtractor Class used to extractWeightedSpanTerm
s from aQuery
based on whetherTerm
s from theQuery
are contained in a suppliedTokenStream
.WeightedTerm Lightweight class to hold term and a weight value used for scoring this term -
Exception Summary Exception Description InvalidTokenOffsetsException Exception thrown if TokenStream Tokens are incompatible with provided text