lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Khanzode <sandeep_khanz...@yahoo.com.INVALID>
Subject Re: BitSet in Filters
Date Tue, 12 Aug 2014 16:08:01 GMT
Hi Erick,

I have mentioned everything that is relevant, I believe :).

However, just to give more background: Assume documents of the order of more than 300 million,
and multiple concurrent users running search. I may front Lucene with ElasticSearch, and ES
basically calls Lucene TermFilters. My filters are broad in nature, so you can take it that
any time I filter on a tag, it would run into, easily, millions of documents to be accepted
in the filter.

The only filter that uses a BitSet works with Document Ids in Lucene. I would have wanted
this bitset approach to work on some other regular numeric long field so that we can scale
which does not seem likely if I have to use an ArrayList of Longs for TermFilters.

Hope that makes the scenario more clear. Please let me know your thoughts.
 
-----------------------
Thanks n Regards,
Sandeep Ramesh Khanzode


On Tuesday, August 12, 2014 8:41 PM, Erick Erickson <erickerickson@gmail.com> wrote:
 


bq: Unless, I can cache these filters in memory, the cost of constructing this filter at
run time per query is not practical

Why do you say that? Do you have evidence? Because lots and lots of Solr installations do
exactly this and they run fine.

So I suspect there's something you're not telling us about your setup. Are you, say, soft
committing often? Do you have autowarming specified? 

You're not going to be able to keep your filters based on some other field in the document.
Internally, Lucene uses the internal doc ID as an index into the bitset. That's baked in to
very low levels and isn't going to change AFAIK.

Best,
Erick



On Mon, Aug 11, 2014 at 11:53 PM, Sandeep Khanzode <sandeep_khanzode@yahoo.com.invalid>
wrote:

Hi,
> 
>The current usage of BitSets in filters in Lucene is limited to applying only on docIDs
i.e. I can only construct a filter out of a BitSet if I have the DocumentIDs handy.
>
>However, with every update/delete i.e. CRUD modification, these will change, and I have
to again redo the whole process to fetch the latest docIDs. 
>
>Assume a scenario where I need to tag millions of documents with a tag like "Finance",
"IT", "Legal", etc.
>
>Unless, I can cache these filters in memory, the cost of constructing this filter at run
time per query is not practical. If I could map the documents to a numeric long identifier
and put them in a BitMap, I could then cache them because the size reduces drastically. However,
I cannot use this numeric long identifier in Lucene filters because it is not a docID but
another regular field.
>
>Please help with this scenario. Thanks,
>
>-----------------------
>Thanks n Regards,
>Sandeep Ramesh Khanzode
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message