lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Istvan Soos <istvan.s...@gmail.com>
Subject Re: best practice on too many files vs IO overhead
Date Fri, 27 Nov 2009 11:11:33 GMT
You were right, my bad...

I have an async reader closing on a scheduled basis (after the writer
refreshes the index, to not interrupt the ongoing searches), but while
I've setup the scheduling for my first two index, I've forgotten it in
my third... oh dear...

Thanks anyway the info, it was useful indeed.
Regards,
   Istvan

On Fri, Nov 27, 2009 at 12:02 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> If in fact you are using CFS (it is the default), and your OS is
> letting you use 10240 descriptors, and you haven't changed the
> mergeFactor, then something is seriously wrong.  I would triple check
> that all readers are being closed.
>
> Or... if you list the index directory, how many files do you see?
>
> Mike
>
> On Fri, Nov 27, 2009 at 5:48 AM, Istvan Soos <istvan.soos@gmail.com> wrote:
>> On Fri, Nov 27, 2009 at 11:37 AM, Michael McCandless
>> <lucene@mikemccandless.com> wrote:
>>> Are you sure you're closing all readers that you're opening?
>>
>> Absolutely. :) (okay, never say this, but I had bugz because of this
>> previously so I'm pretty sure that one is ok).
>>
>>> It's surprising with normal usage of Lucene that you'd run out of
>>> descriptors, with its default mergeFactor (have you increased the
>>> mergeFactor)?
>>
>> Default merge factor. (on Mac, the default maxfiles is 256, however
>> I've run out of descriptors event at 10240, if I hadn't called
>> optimize).
>>
>>> You can also enable compound file, which uses far fewer file
>>> descriptors, at some cost to indexing performance.
>>
>> I thought this is the default but I'll check...
>>
>>> Also, a partial optimize (ie optimize(N)) does less IO but still
>>> substantially reduces segment count of the index.
>>
>> I wasn't aware of this, thanks, I'll try it!
>>
>> Regards,
>>   Istvan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message