lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Wang <john.w...@gmail.com>
Subject Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Date Sun, 02 May 2010 23:12:47 GMT
12 concurrent search threads in a stress environment. (with about 5M doc
index)

-John

On Sun, May 2, 2010 at 3:13 PM, Mark Miller <markrmiller@gmail.com> wrote:

> Perfectly safe to use SimpleFSDirectory. When you measure the performance,
> are you measuring a lot of concurrent requests? That's where you should see
> the win.
>
> - Mark
>
>
> On 5/1/10 6:38 PM, John Wang wrote:
>
>> We are seeing this issue as well in your production. (using Zoie on top
>> of lucene 2.9.1)
>> After some performance comparisons, we do NOT see performance gain with
>> NIO, rather these nasty ClosedChannelExceptions.
>>
>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>> many different things. From what we seen, they are not related to
>> NIOFSDirectory.
>>
>> Our solution is to avoid calling FSDirectory.open(), instead just call
>> new SimpleFSDirectory(). Is this safe?
>>
>> -John
>>
>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>> <mailto:markrmiller@gmail.com>> wrote:
>>
>>    Perhaps - one of the things they are supposed to be addressing is
>>    extendability.
>>
>>    nio2 does have FileSystemProvider, which would actually allow you to
>>    create a custom channel !
>>
>>    I have not dug in enough to know much more than that though.
>>
>>    *But*, another really interesting thing is that in Java 7,
>>    FileDescriptors are ref counted ! (though users can't inc/dec).
>>
>>    But, FileInputStream and OutputStream have a new constructor that takes
>>    a FileDescriptor.
>>
>>    So possibly, you could just make one that sits around to keep the
>>    FileDescriptor valid, and get your channel off
>>    FileInputStream/FileOutputStream?
>>
>>    And then if it goes down, make a new one using the FileDescriptor which
>>    was not actually closed because there was a still a ref to it.
>>
>>    Possibly .... ;)
>>
>>    Michael McCandless wrote:
>>     > Does anyone know if nio2 has improved this...?
>>     >
>>     > Mike
>>     >
>>     > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>>     > <jason.rutherglen@gmail.com <mailto:jason.rutherglen@gmail.com>>
>>
>>    wrote:
>>     >
>>     >> Defaulting NIOFSDir could account for some of the recent speed
>>     >> improvements users have been reporting in Lucene 2.9.  So
>>    removing it
>>     >> as a default could reverse those and people could then report
>> Lucene
>>     >> 3.X has slowed...
>>     >>
>>     >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>>     >> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>
>>
>>    wrote:
>>     >>
>>     >>> Bummer.
>>     >>>
>>     >>> So the only viable workarounds are 1) don't use
>>    Thread.interrupt (nor,
>>     >>> things like Future.cancel, which in turn use Thread.interrupt)
>> with
>>     >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>>    app must
>>     >>> make a deletion policy that keeps a commit alive if any reader is
>>     >>> using it.  Or, 3) don't use NIOFSDir!
>>     >>>
>>     >>> Mike
>>     >>>
>>     >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>     >>> <simon.willnauer@googlemail.com
>>    <mailto:simon.willnauer@googlemail.com>> wrote:
>>     >>>
>>     >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>     >>>> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>
>>
>>    wrote:
>>     >>>>
>>     >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>>    <uwe@thetaphi.de <mailto:uwe@thetaphi.de>> wrote:
>>     >>>>>
>>     >>>>>
>>     >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>>    was not really correct:
>>     >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that
opens
>>    the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>>    until the file is closed (and also the channel).
>>     >>>>>>
>>     >>>>>> So it's really simple to fix in my opinion, just call
>>    getChannel() again on this exception. Because the RAF should still
>>    be open?
>>     >>>>>>
>>     >>>> Short answer:
>>     >>>>  public final FileChannel getChannel() {
>>     >>>>        synchronized (this) {
>>     >>>>            if (channel == null)
>>     >>>>                channel = FileChannelImpl.open(fd, true, rw,
>> this);
>>     >>>>            return channel;
>>     >>>>        }
>>     >>>>    }
>>     >>>>
>>     >>>> this is not gonna work I tried it before. The RandomAccessFile
>>    buffers
>>     >>>> the channel!!
>>     >>>>
>>     >>>> simon
>>     >>>>
>>     >>>>> I think we need a definitive answer on what happens to the
>>    RAF when
>>     >>>>> the FileChannel was closed by Thread.Interrupt.  Simon can
>>    you test
>>     >>>>> this?
>>     >>>>>
>>     >>>>> Mike
>>     >>>>>
>>     >>>>>
>>    ---------------------------------------------------------------------
>>     >>>>> To unsubscribe, e-mail:
>>    java-dev-unsubscribe@lucene.apache.org
>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>
>>     >>>>> For additional commands, e-mail:
>>    java-dev-help@lucene.apache.org <mailto:
>> java-dev-help@lucene.apache.org>
>>
>>     >>>>>
>>     >>>>>
>>     >>>>>
>>     >>>>
>>    ---------------------------------------------------------------------
>>     >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>
>>     >>>> For additional commands, e-mail:
>>    java-dev-help@lucene.apache.org <mailto:
>> java-dev-help@lucene.apache.org>
>>
>>     >>>>
>>     >>>>
>>     >>>>
>>     >>>
>>    ---------------------------------------------------------------------
>>     >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>
>>     >>> For additional commands, e-mail:
>>    java-dev-help@lucene.apache.org <mailto:
>> java-dev-help@lucene.apache.org>
>>
>>     >>>
>>     >>>
>>     >>>
>>     >>
>>    ---------------------------------------------------------------------
>>     >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>
>>     >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>    <mailto:java-dev-help@lucene.apache.org>
>>
>>     >>
>>     >>
>>     >>
>>     >
>>     >
>> ---------------------------------------------------------------------
>>     > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>
>>     > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>    <mailto:java-dev-help@lucene.apache.org>
>>
>>     >
>>     >
>>
>>
>>    --
>>    - Mark
>>
>>    http://www.lucidimagination.com
>>
>>
>>
>>
>>    ---------------------------------------------------------------------
>>    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>
>>    For additional commands, e-mail: java-dev-help@lucene.apache.org
>>    <mailto:java-dev-help@lucene.apache.org>
>>
>>
>>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Mime
View raw message