lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miles Barr <mi...@magpie.net>
Subject Re: drill-down heuristics WAS: Where to find drill-down examples (source code)
Date Mon, 24 Jul 2006 11:00:58 GMT
On Monday 24 July 2006 08:17, Martin Braun wrote:
> I think I didn't explain my Problem good enough.
>
> The harder problem for me is how to get the proposals for the
> refinement?  I have a date-range of 16xx to now, for about 4 bn. docs.
> So the number of found documents could be quite large. But the
> distribution of the dates could be very different form one query to
> another. I hope there is a better way than to collect all dates with
> HitIterator, and do statistics on the data?
>
> Is there something that could be done while indexing?
> What would be a high-performance heuristic?
>
> The same problem with other categories like the author: how to find good
> proposals for a given result set?

That's a lot trickier and there might be others on the list who can give a 
better answer. I think what you need to do is to extend HitCollector:

http://lucene.apache.org/java/docs/api/org/apache/lucene/search/HitCollector.html

Your implementation will keep a count of dates and categories that the results 
are in. Then you can use this information to build up your refinements. The 
problem with this approach is that HitCollector#collect only provides you 
with the document number, not the document itself, hence you still have to 
load the document to find out its date and category, this will be slow for 
large result sets.

You might want to keep this metadata in another store so you can quickly look 
up the date and category based on the document number. This avoids having to 
load all the documents from Lucene, which is an expensive operation. The 
downside with this approach is that the document number changes when you 
optimise your index, hence you'll have to rebuild your metadata store each 
time you optimise.



Miles

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message