lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Wang <john.w...@gmail.com>
Subject Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Date Mon, 03 May 2010 03:53:25 GMT
Solaris and MacOs

On Sun, May 2, 2010 at 4:47 PM, Mark Miller <markrmiller@gmail.com> wrote:

> Interesting ... whats the OS?
>
>
> On Sun, May 2, 2010 at 7:12 PM, John Wang <john.wang@gmail.com> wrote:
>
>> 12 concurrent search threads in a stress environment. (with about 5M doc
>> index)
>>
>> -John
>>
>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <markrmiller@gmail.com>wrote:
>>
>>> Perfectly safe to use SimpleFSDirectory. When you measure the
>>> performance, are you measuring a lot of concurrent requests? That's where
>>> you should see the win.
>>>
>>> - Mark
>>>
>>>
>>> On 5/1/10 6:38 PM, John Wang wrote:
>>>
>>>> We are seeing this issue as well in your production. (using Zoie on top
>>>> of lucene 2.9.1)
>>>> After some performance comparisons, we do NOT see performance gain with
>>>> NIO, rather these nasty ClosedChannelExceptions.
>>>>
>>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>>>> many different things. From what we seen, they are not related to
>>>> NIOFSDirectory.
>>>>
>>>> Our solution is to avoid calling FSDirectory.open(), instead just call
>>>> new SimpleFSDirectory(). Is this safe?
>>>>
>>>> -John
>>>>
>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>>>> <mailto:markrmiller@gmail.com>> wrote:
>>>>
>>>>    Perhaps - one of the things they are supposed to be addressing is
>>>>    extendability.
>>>>
>>>>    nio2 does have FileSystemProvider, which would actually allow you to
>>>>    create a custom channel !
>>>>
>>>>    I have not dug in enough to know much more than that though.
>>>>
>>>>    *But*, another really interesting thing is that in Java 7,
>>>>    FileDescriptors are ref counted ! (though users can't inc/dec).
>>>>
>>>>    But, FileInputStream and OutputStream have a new constructor that
>>>> takes
>>>>    a FileDescriptor.
>>>>
>>>>    So possibly, you could just make one that sits around to keep the
>>>>    FileDescriptor valid, and get your channel off
>>>>    FileInputStream/FileOutputStream?
>>>>
>>>>    And then if it goes down, make a new one using the FileDescriptor
>>>> which
>>>>    was not actually closed because there was a still a ref to it.
>>>>
>>>>    Possibly .... ;)
>>>>
>>>>    Michael McCandless wrote:
>>>>     > Does anyone know if nio2 has improved this...?
>>>>     >
>>>>     > Mike
>>>>     >
>>>>     > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>>>>     > <jason.rutherglen@gmail.com <mailto:jason.rutherglen@gmail.com>>
>>>>
>>>>    wrote:
>>>>     >
>>>>     >> Defaulting NIOFSDir could account for some of the recent speed
>>>>     >> improvements users have been reporting in Lucene 2.9.  So
>>>>    removing it
>>>>     >> as a default could reverse those and people could then report
>>>> Lucene
>>>>     >> 3.X has slowed...
>>>>     >>
>>>>     >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>>>>     >> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>
>>>>
>>>>    wrote:
>>>>     >>
>>>>     >>> Bummer.
>>>>     >>>
>>>>     >>> So the only viable workarounds are 1) don't use
>>>>    Thread.interrupt (nor,
>>>>     >>> things like Future.cancel, which in turn use Thread.interrupt)
>>>> with
>>>>     >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND
the
>>>>    app must
>>>>     >>> make a deletion policy that keeps a commit alive if any
reader
>>>> is
>>>>     >>> using it.  Or, 3) don't use NIOFSDir!
>>>>     >>>
>>>>     >>> Mike
>>>>     >>>
>>>>     >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>>>     >>> <simon.willnauer@googlemail.com
>>>>    <mailto:simon.willnauer@googlemail.com>> wrote:
>>>>     >>>
>>>>     >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>>>     >>>> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>
>>>>
>>>>    wrote:
>>>>     >>>>
>>>>     >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>>>>    <uwe@thetaphi.de <mailto:uwe@thetaphi.de>> wrote:
>>>>     >>>>>
>>>>     >>>>>
>>>>     >>>>>> So I checked the code of NIOFSIndexInput, my
last comment
>>>>    was not really correct:
>>>>     >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and
that opens
>>>>    the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>>>>    until the file is closed (and also the channel).
>>>>     >>>>>>
>>>>     >>>>>> So it's really simple to fix in my opinion,
just call
>>>>    getChannel() again on this exception. Because the RAF should still
>>>>    be open?
>>>>     >>>>>>
>>>>     >>>> Short answer:
>>>>     >>>>  public final FileChannel getChannel() {
>>>>     >>>>        synchronized (this) {
>>>>     >>>>            if (channel == null)
>>>>     >>>>                channel = FileChannelImpl.open(fd, true,
rw,
>>>> this);
>>>>     >>>>            return channel;
>>>>     >>>>        }
>>>>     >>>>    }
>>>>     >>>>
>>>>     >>>> this is not gonna work I tried it before. The RandomAccessFile
>>>>    buffers
>>>>     >>>> the channel!!
>>>>     >>>>
>>>>     >>>> simon
>>>>     >>>>
>>>>     >>>>> I think we need a definitive answer on what happens
to the
>>>>    RAF when
>>>>     >>>>> the FileChannel was closed by Thread.Interrupt.
 Simon can
>>>>    you test
>>>>     >>>>> this?
>>>>     >>>>>
>>>>     >>>>> Mike
>>>>     >>>>>
>>>>     >>>>>
>>>>    ---------------------------------------------------------------------
>>>>     >>>>> To unsubscribe, e-mail:
>>>>    java-dev-unsubscribe@lucene.apache.org
>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>
>>>>     >>>>> For additional commands, e-mail:
>>>>    java-dev-help@lucene.apache.org <mailto:
>>>> java-dev-help@lucene.apache.org>
>>>>
>>>>     >>>>>
>>>>     >>>>>
>>>>     >>>>>
>>>>     >>>>
>>>>    ---------------------------------------------------------------------
>>>>     >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>
>>>>     >>>> For additional commands, e-mail:
>>>>    java-dev-help@lucene.apache.org <mailto:
>>>> java-dev-help@lucene.apache.org>
>>>>
>>>>     >>>>
>>>>     >>>>
>>>>     >>>>
>>>>     >>>
>>>>    ---------------------------------------------------------------------
>>>>     >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>
>>>>     >>> For additional commands, e-mail:
>>>>    java-dev-help@lucene.apache.org <mailto:
>>>> java-dev-help@lucene.apache.org>
>>>>
>>>>     >>>
>>>>     >>>
>>>>     >>>
>>>>     >>
>>>>    ---------------------------------------------------------------------
>>>>     >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>
>>>>     >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>    <mailto:java-dev-help@lucene.apache.org>
>>>>
>>>>     >>
>>>>     >>
>>>>     >>
>>>>     >
>>>>     >
>>>> ---------------------------------------------------------------------
>>>>     > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>
>>>>     > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>    <mailto:java-dev-help@lucene.apache.org>
>>>>
>>>>     >
>>>>     >
>>>>
>>>>
>>>>    --
>>>>    - Mark
>>>>
>>>>    http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>>
>>>>    ---------------------------------------------------------------------
>>>>    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>
>>>>    For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>    <mailto:java-dev-help@lucene.apache.org>
>>>>
>>>>
>>>>
>>>
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>
>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>

Mime
View raw message