lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomislav Poljak <tpol...@gmail.com>
Subject Re: MultiPhraseQuery slowing down over time in Lucene 3.1
Date Tue, 03 May 2011 11:43:40 GMT
Hi,

2011/5/3 Michael McCandless <lucene@mikemccandless.com>:
> I feel like we are back to Basic ;)
>
> If you keep running line 40 over and over on the same memory index, do
> you see a slowdown?

Yes. I've tested running same query list (~3,5 k queries) on the same
MemoryIndex instance and after a while iterations get slower and
slower. Same thing happens when running queries on the same instance
of RAMDir based index holding only one doc. But, if I remove
MultiPhraseQuery type of queries from the query list then speed of
execution is the same, meaning execution time for other queries is
constant and it doesn't grow over time (as it would be expected).

I've tried to run tests with the latest 3.1 branch as Uwe suggested
(checkout and built today) and slowdown is still present when
MultiPhraseQuery type of queries are included (not removed from the
query list).

Tomislav

>
> Mike
>
> http://blog.mikemccandless.com
>
> On Mon, May 2, 2011 at 1:19 PM, Otis Gospodnetic
> <otis_gospodnetic@yahoo.com> wrote:
>> Hi,
>>
>> I think this describes what's going on:
>>
>> 10 load N stored queries
>> 20 parse N stored queries, keep them in some List forever
>> 30 for each incoming document create a new MemoryIndex instance "mi"
>> 40 for query 1 to N do mi.search(query)
>>
>> Over time this step 40 takes longer and longer and longer -- if some of the
>> queries are MultiPhraseQueries.  This is even with with mergeSort being used in
>> MultiPhraseQuery.
>>
>> Otis
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> Lucene ecosystem search :: http://search-lucene.com/
>>
>>
>>
>> ----- Original Message ----
>>> From: Michael McCandless <lucene@mikemccandless.com>
>>> To: java-user@lucene.apache.org
>>> Sent: Mon, May 2, 2011 12:15:40 PM
>>> Subject: Re: MultiPhraseQuery slowing down over time in Lucene 3.1
>>>
>>> By "slowing down over time" do you mean you use the same index (no new
>>> docs  added) yet running the same MPQ over and over you see it taking
>>> longer to  execute over time?
>>>
>>> Mike
>>>
>>> http://blog.mikemccandless.com
>>>
>>> On Mon, May 2, 2011 at  12:00 PM, Tomislav Poljak <tpoljak@gmail.com>
wrote:
>>> >  Hi,
>>> > after running tests on both MemoryIndex and RAMDirectory based  index
>>> > in Lucene 3.1, seems MultiPhraseQueries are slowing down over  time
>>> > (each iteration of executing the same MultiPhraseQueries on the  same
>>> > doc, seems to require more and more execution time). Are there  any
>>> > existing/known issues related to the MultiPhraseQuery in Lucene  3.1
>>> > which could lead to this performance drop?
>>> >
>>> >  Tomislav
>>> >
>>> >  ---------------------------------------------------------------------
>>> > To  unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >  For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To  unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For  additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message