lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Serebrennikov <>
Subject Re: suggestion for a CustomDirectory
Date Fri, 05 Dec 2003 23:41:19 GMT
Doug Cutting wrote:

> fp235-5 wrote:
>> Dmitry's patch improved this
>> part a lot and in my case reduced by 10-15% the overall time. Sadly 
>> it has never
>> been included  in the source and could have been useful for all kind 
>> of users.
> If you read the thread associated with that patch you'll see that it 
> made other things slower, because it removed another optimization.  So 
> while it would have helped some folks, it would have hurt others.

I think that the removal of that other optimization was more due to my 
not understanding it well enough, rather than an integral part of 
reducing term creation. It's been a while though, so I don't know for 
sure any more. My recollection is that both could be accommodated with 
some work. In our environment/application we do not hit this other case 
and so have been happily running with the reduced Term object creation 
ever since that thread.

I think for these GC effects to really start to show up one has to have 
all of the following:
    - extensive TermEnum scanning (as in non-startsWith wildcard 
queries, or with other application code)
    - multi-CPU machine
    - large number of concurrent queries
    - large number of terms in the indexes

 From what I remember of my tests, I found that scanning the term enums 
for matching terms caused lots of term objects to be created and 
discarded, which required excessive GC, which had severe performance 
consequences on multi-CPU machines (because the CPUs produce garbage in 
parallel but GC can only use one CPU to collect it). The multi-CPU issue 
is supposed to be fixed in Java 1.4, but I had not experimented with it 
yet, so I don't know if that is really effective.


> Doug
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message