lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mark harwood <>
Subject Re: Lucene Queries Over User-Editable Dynamic Categories of Documents
Date Wed, 24 Oct 2007 13:02:34 GMT
If you were talking about a reasonably stable dataset e.g. products in a catalogue that would
be manageable in Lucene because the volume of updates is comparatively low (one set of categorisations
maintained by the site owner e.g. something like Cnet where Solr/faceted search came from).
If you are talking about lots of users with their own categories/tags (I'm thinking of del.ici.ous/Flickr
model) then the update volumes are likely to be much higher and therefore could be a problem
when maintaining a single Lucene index because of the need to constantly close/reopen readers/writers.

Managing multiple per-user indexes may be an option but I've never tried that - that would
require more disk, RAM and management.

As for the original suggestion: I have used thousands of primary-key terms in a terms filter
before now (i.e. terms with only one doc) and was surprised at the speed. I can't recall exact
stats so try it yourself.

----- Original Message ----
From: lucene user <>
Sent: Wednesday, 24 October, 2007 1:12:32 PM
Subject: Re: Lucene Queries Over User-Editable Dynamic Categories of Documents

Thanks very much. How large can my end-user's catigories grow before
implementation you have outlined will start to bog down? If my users
thousands of items categorized, would you still recommend using a term
filter in this way? Tens of thousands? What is a realistic max? Is
another idea that works for even larger numbers? Frankly, we don't yet
understand how our users will use the system in the long run. When you
done stuff like this, how large have the term filters grown?

Would it EVER make sense to maintain the end user's catigories in some
of Lucene data structure? If so, what data structure?

Would it EVER be wise to keep the end-user catigories in Lucene? If so,
and how?

What are other realistic options for implementing user categorization

When solr talks about "faceted searching," this isn't what they mean,
 is it?
They say "Faceted Searching based on unique field values and explicit
queries" and I'm looking to find what they mean and not getting clear.


On 10/24/07, mark harwood <> wrote:
> Given the volatility in the set membership I'd be tempted to keep
> grouping info in a database rather than doing the
> dance in Lucene before you can see any updates. (I suspect this is
> reason you've opted not to keep the info in Lucene).
> You can pull a user's list of a hundred or so terms out of the
> (typically primary keys) and add them as a TermsFilter to your Lucene
> queries.
> I've found that using this approach can be pretty fast even with a
> list of filter terms - it was a while ago so I can't quote stats,
> need to try it for yourself.
> Caching these filters may prove useful but if it's a big dataset
> don't sound like a memory-efficient form of storing these lists as it
> like they'll be sparsely populated.
> You may be interested in the more memory-efficient options such as
> SortedVIntList here:
> Without taking the whole of that patch on board you could have a
> strategy based on this pseudocode:
> getFilter(Set primaryKeys, IndexReader reader)
> {
>    TermsFilter tf= new TermsFilter()
>    for all primaryKeys:
>        tf.addTerm(primaryKey)
>   BitSet bits;
>   SortedVIntList cached=lruCachedMap.get(tf);
>   if(cached==null)
>         bits=tf.bits(reader)
>         lruCachedMap.put(tf, convertBitsToSortedVIntList(bits))
>   else
>         bits=convertSortedVIntListToBits(bits)
>   return new Filter()
>        {
>                  BitSet bits(IndexReader reader)
>                  {
>                      return bits;
>                  }
>        };
> }
> On a bit of a lucene-dev tangent, I think the above code has the
> of an optimisation to CachingWrapperFilter - it could choose to cache
> SortedVIntLists or BitSets depending on the sparseness of the set and
> transparently handles any required conversions.
> ----- Original Message ----
> From: lucene user <>
> To:
> Sent: Wednesday, 24 October, 2007 7:18:10 AM
> Subject: Lucene Queries Over User-Editable Dynamic Categories of
> Folks!
> We are building a web-based multi-user system. Users of our system
> able
> to categorize items that they have found into groups of related
> documents.
> We would like users to be able to search these document groups and
> rapidly
> find matches. Each user might have ten of these categories and might
> have
> perhaps a few hundred documents in each. These categories might be
> highly
> dynamic, with users adding and deleting documents from these
> many
> times a day. How might we use Lucene to perform searches limited to
> these
> very dynamic and end-user editable categories? Any ideas for how we
> might do
> this efficiently?
> If all the data were in a SQL database, we could run a subquery that
> returned the IDs of the items in categories and use that to limit the
> results of the super query.
> Currently we do not plan to maintain the information about the
> end-user's
> categories in the Lucene index at all, or not in a big, main Lucene
> index
> anyway.
> What our the reasonable options for handling this? What are the
> performance
> implications of various choices?
> Thanks!

Yahoo! Answers - Got a question? Someone out there knows the answer. Try it

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message