lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Serebrennikov <dmit...@earthlink.net>
Subject Re: suggestion for a CustomDirectory
Date Fri, 05 Dec 2003 23:41:19 GMT
Doug Cutting wrote:

> fp235-5 wrote:
>
>> Dmitry's patch improved this
>> part a lot and in my case reduced by 10-15% the overall time. Sadly 
>> it has never
>> been included  in the source and could have been useful for all kind 
>> of users.
>
>
> If you read the thread associated with that patch you'll see that it 
> made other things slower, because it removed another optimization.  So 
> while it would have helped some folks, it would have hurt others.

I think that the removal of that other optimization was more due to my 
not understanding it well enough, rather than an integral part of 
reducing term creation. It's been a while though, so I don't know for 
sure any more. My recollection is that both could be accommodated with 
some work. In our environment/application we do not hit this other case 
and so have been happily running with the reduced Term object creation 
ever since that thread.

I think for these GC effects to really start to show up one has to have 
all of the following:
    - extensive TermEnum scanning (as in non-startsWith wildcard 
queries, or with other application code)
    - multi-CPU machine
    - large number of concurrent queries
    - large number of terms in the indexes

 From what I remember of my tests, I found that scanning the term enums 
for matching terms caused lots of term objects to be created and 
discarded, which required excessive GC, which had severe performance 
consequences on multi-CPU machines (because the CPUs produce garbage in 
parallel but GC can only use one CPU to collect it). The multi-CPU issue 
is supposed to be fixed in Java 1.4, but I had not experimented with it 
yet, so I don't know if that is really effective.

Dmitry

>
> Doug
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message