lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mark harwood <>
Subject Re: Highlighter that works with phrase and span queries
Date Wed, 27 Jun 2007 15:17:50 GMT
>>you would still have the major problem of which matches do you keep information for

Yes, doing this efficiently is the main issue. Some vague thoughts I had:
1) A special HighlightObserverQuery could wrap any query and use it's rewrite method to further
wrap child component queries if necessary.
2) A ThreadLocal could be used to contain low-level match info generated by child query components
e.g. position info of phrase/span queries (maybe generatable by a HighlightingIndexReader
wrapper which observed TermPositions access)
3) For each call to on the top level query, the HighlightObserver class would
check to see if the doc was a "keeper" (i.e. it's score places it in the required top "n"
docs PriorityQueue) and if so, would retain a copy of all the transient match info currently
held in the ThreadLocal for this doc and associate it with the new TopDoc object placed in
the top docs PriorityQueue.

This approach tries hard not to require changes to existing Query/scorer classes by using
wrappers/ThreadLocals and would only hold low-level match highlighting info for N documents
where "N" is the maximum number of results to be returned. 
However there are likely to be many detailed complications with implementing this. I haven't
pursued this train of thought further because the main killer is likely to be the performance
overhead from all the unnecessary object creation when generating match info objects for documents
that don't make the final selection anyway. That and the cost of synchronization around ThreadLocal

I think we're right to stick with the existing highlighting approach of searching for the
top N docs then re-considering the basis of the match for just these few docs.


----- Original Message ----
From: Mark Miller <>
Sent: Wednesday, 27 June, 2007 12:59:21 PM
Subject: Re: Highlighter that works with phrase and span queries

markharw00d wrote:
> I was thinking along the lines of wrapping some core classes such as 
> IndexReader to somehow observe the query matching process and deduce 
> from that what to highlight (avoiding the need for MemoryIndex)  but 
> I'm not sure that is viable. It would be nice to get some more match 
> info out of the main query logic as it runs to aid highlighting rather 
> than reverse engineering the basis of a match after the event. 

I have been thinking about a way to pursue this, and it does not seem 
clear that there is a nice solution. Even if you could wrap Querys or 
other classes to observe matched tokens (non trivial since a Query is 
only concerned if it matches a doc, not which tokens it matches at which 
positions), you would still have the major problem of which matches do 
you keep information for. It does not seem practical to save all of the 
information to highlight *any* doc after a search and it also seems 
unlikely that you would know which docs you wanted to highlight before 
the search. The only compromise that I can see is maybe just storing 
info to highlight the first n docs, but even here, while the scoring is 
occurring you do not yet know the return order. Also, there is probably 
little value in knowing which Tokens were matches for highlighting 
unless you have stored offsets as well.

Unless someone has any suggestions on how to accomplish this, I think 
time would be better spent improving the existing Highlighter framework.

Perhaps Ronnie's Highlighter should be added as an alternate Highlighter 
that is less feature rich but much faster on large docs. It looks to me 
like there is unlikely to be a faster Highlighting method for simple 
non-position aware highlighting.

- Mark

To unsubscribe, e-mail:
For additional commands, e-mail:

Yahoo! Answers - Got a question? Someone out there knows the answer. Try it

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message