Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Subject: Re: Lucene TermsFilter lookup slow
To: java-user@lucene.apache.org
References: <55C5F6E2.2090804@stimulussoft.com>
 <CAL8PwkYhNqnoSO=GRuGXohqkJprpRt7gSpSmU6izbJg=5hDOFA@mail.gmail.com>
From: jamie <jamie@stimulussoft.com>
Message-ID: <55C6FE8D.2080102@stimulussoft.com>
Date: Sun, 9 Aug 2015 09:17:33 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0)
 Gecko/20100101 Thunderbird/38.1.0
MIME-Version: 1.0
In-Reply-To: 
 <CAL8PwkYhNqnoSO=GRuGXohqkJprpRt7gSpSmU6izbJg=5hDOFA@mail.gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit

Mike

Thank you kindly for the reply. I am using Lucene v4.10.4. Are the 
optimization you refer to, available in this version?

We haven't yet upgraded to Lucene 5 as there appear to be many API changes.

Jamie

On 2015/08/08 5:13 PM, Michael McCandless wrote:
> Which version of Lucene are you using?  Newer versions have optimized
> the "primary key" use case somewhat...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Sat, Aug 8, 2015 at 8:32 AM, jamie <jamie@stimulussoft.com> wrote:
>> Greetings
>>
>> Our app primarily uses Lucene for its intended purpose i.e. to search across
>> large amounts of unstructured text. However, recently our requirement
>> expanded to perform look-ups on specific documents in the index based on
>> associated custom defined unique keys. For our purposes, a unique key is the
>> string representation of a 128 bit murmur hash, stored in a Lucene field
>> named uid.  We are currently using the TermsFilter to lookup Documents in
>> the Lucene index as follows:
>>
>> List<Term> terms = new LinkedList<>();
>>              for (String id : ids) {
>>                  terms.add(new Term("uid", id));
>> }
>> TermsFilter idFilter = new TermsFilter(terms);
>> ... search logic...
>>
>> At any time we may need to lookup say a couple of thousand documents. Our
>> problem is one of performance. On very large indexes with 30 million records
>> or more, the lookup can be excruciatingly slow. At this stage, its not
>> practical for us to move the data over to fit for purpose database, nor
>> change the uid field to a numeric type. I fully appreciate the fact that
>> Lucene is not designed to be a database, however, is there anything we can
>> do to improve the performance of these look-ups?
>>
>> Much appreciate
>>
>> Jamie
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org