lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: best practice on too many files vs IO overhead
Date Fri, 27 Nov 2009 11:12:39 GMT
Phew :)  Thanks for bringing closure!

Mike

On Fri, Nov 27, 2009 at 6:02 AM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> If in fact you are using CFS (it is the default), and your OS is
> letting you use 10240 descriptors, and you haven't changed the
> mergeFactor, then something is seriously wrong.  I would triple check
> that all readers are being closed.
>
> Or... if you list the index directory, how many files do you see?
>
> Mike
>
> On Fri, Nov 27, 2009 at 5:48 AM, Istvan Soos <istvan.soos@gmail.com> wrote:
>> On Fri, Nov 27, 2009 at 11:37 AM, Michael McCandless
>> <lucene@mikemccandless.com> wrote:
>>> Are you sure you're closing all readers that you're opening?
>>
>> Absolutely. :) (okay, never say this, but I had bugz because of this
>> previously so I'm pretty sure that one is ok).
>>
>>> It's surprising with normal usage of Lucene that you'd run out of
>>> descriptors, with its default mergeFactor (have you increased the
>>> mergeFactor)?
>>
>> Default merge factor. (on Mac, the default maxfiles is 256, however
>> I've run out of descriptors event at 10240, if I hadn't called
>> optimize).
>>
>>> You can also enable compound file, which uses far fewer file
>>> descriptors, at some cost to indexing performance.
>>
>> I thought this is the default but I'll check...
>>
>>> Also, a partial optimize (ie optimize(N)) does less IO but still
>>> substantially reduces segment count of the index.
>>
>> I wasn't aware of this, thanks, I'll try it!
>>
>> Regards,
>>   Istvan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message