lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Date Sun, 02 May 2010 23:47:34 GMT
Interesting ... whats the OS?

On Sun, May 2, 2010 at 7:12 PM, John Wang <john.wang@gmail.com> wrote:

> 12 concurrent search threads in a stress environment. (with about 5M doc
> index)
>
> -John
>
> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <markrmiller@gmail.com> wrote:
>
>> Perfectly safe to use SimpleFSDirectory. When you measure the performance,
>> are you measuring a lot of concurrent requests? That's where you should see
>> the win.
>>
>> - Mark
>>
>>
>> On 5/1/10 6:38 PM, John Wang wrote:
>>
>>> We are seeing this issue as well in your production. (using Zoie on top
>>> of lucene 2.9.1)
>>> After some performance comparisons, we do NOT see performance gain with
>>> NIO, rather these nasty ClosedChannelExceptions.
>>>
>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>>> many different things. From what we seen, they are not related to
>>> NIOFSDirectory.
>>>
>>> Our solution is to avoid calling FSDirectory.open(), instead just call
>>> new SimpleFSDirectory(). Is this safe?
>>>
>>> -John
>>>
>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>>> <mailto:markrmiller@gmail.com>> wrote:
>>>
>>>    Perhaps - one of the things they are supposed to be addressing is
>>>    extendability.
>>>
>>>    nio2 does have FileSystemProvider, which would actually allow you to
>>>    create a custom channel !
>>>
>>>    I have not dug in enough to know much more than that though.
>>>
>>>    *But*, another really interesting thing is that in Java 7,
>>>    FileDescriptors are ref counted ! (though users can't inc/dec).
>>>
>>>    But, FileInputStream and OutputStream have a new constructor that
>>> takes
>>>    a FileDescriptor.
>>>
>>>    So possibly, you could just make one that sits around to keep the
>>>    FileDescriptor valid, and get your channel off
>>>    FileInputStream/FileOutputStream?
>>>
>>>    And then if it goes down, make a new one using the FileDescriptor
>>> which
>>>    was not actually closed because there was a still a ref to it.
>>>
>>>    Possibly .... ;)
>>>
>>>    Michael McCandless wrote:
>>>     > Does anyone know if nio2 has improved this...?
>>>     >
>>>     > Mike
>>>     >
>>>     > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>>>     > <jason.rutherglen@gmail.com <mailto:jason.rutherglen@gmail.com>>
>>>
>>>    wrote:
>>>     >
>>>     >> Defaulting NIOFSDir could account for some of the recent speed
>>>     >> improvements users have been reporting in Lucene 2.9.  So
>>>    removing it
>>>     >> as a default could reverse those and people could then report
>>> Lucene
>>>     >> 3.X has slowed...
>>>     >>
>>>     >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>>>     >> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>
>>>
>>>    wrote:
>>>     >>
>>>     >>> Bummer.
>>>     >>>
>>>     >>> So the only viable workarounds are 1) don't use
>>>    Thread.interrupt (nor,
>>>     >>> things like Future.cancel, which in turn use Thread.interrupt)
>>> with
>>>     >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>>>    app must
>>>     >>> make a deletion policy that keeps a commit alive if any reader
is
>>>     >>> using it.  Or, 3) don't use NIOFSDir!
>>>     >>>
>>>     >>> Mike
>>>     >>>
>>>     >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>>     >>> <simon.willnauer@googlemail.com
>>>    <mailto:simon.willnauer@googlemail.com>> wrote:
>>>     >>>
>>>     >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>>     >>>> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>
>>>
>>>    wrote:
>>>     >>>>
>>>     >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>>>    <uwe@thetaphi.de <mailto:uwe@thetaphi.de>> wrote:
>>>     >>>>>
>>>     >>>>>
>>>     >>>>>> So I checked the code of NIOFSIndexInput, my last
comment
>>>    was not really correct:
>>>     >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that
opens
>>>    the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>>>    until the file is closed (and also the channel).
>>>     >>>>>>
>>>     >>>>>> So it's really simple to fix in my opinion, just
call
>>>    getChannel() again on this exception. Because the RAF should still
>>>    be open?
>>>     >>>>>>
>>>     >>>> Short answer:
>>>     >>>>  public final FileChannel getChannel() {
>>>     >>>>        synchronized (this) {
>>>     >>>>            if (channel == null)
>>>     >>>>                channel = FileChannelImpl.open(fd, true,
rw,
>>> this);
>>>     >>>>            return channel;
>>>     >>>>        }
>>>     >>>>    }
>>>     >>>>
>>>     >>>> this is not gonna work I tried it before. The RandomAccessFile
>>>    buffers
>>>     >>>> the channel!!
>>>     >>>>
>>>     >>>> simon
>>>     >>>>
>>>     >>>>> I think we need a definitive answer on what happens
to the
>>>    RAF when
>>>     >>>>> the FileChannel was closed by Thread.Interrupt.  Simon
can
>>>    you test
>>>     >>>>> this?
>>>     >>>>>
>>>     >>>>> Mike
>>>     >>>>>
>>>     >>>>>
>>>    ---------------------------------------------------------------------
>>>     >>>>> To unsubscribe, e-mail:
>>>    java-dev-unsubscribe@lucene.apache.org
>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>
>>>     >>>>> For additional commands, e-mail:
>>>    java-dev-help@lucene.apache.org <mailto:
>>> java-dev-help@lucene.apache.org>
>>>
>>>     >>>>>
>>>     >>>>>
>>>     >>>>>
>>>     >>>>
>>>    ---------------------------------------------------------------------
>>>     >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>
>>>     >>>> For additional commands, e-mail:
>>>    java-dev-help@lucene.apache.org <mailto:
>>> java-dev-help@lucene.apache.org>
>>>
>>>     >>>>
>>>     >>>>
>>>     >>>>
>>>     >>>
>>>    ---------------------------------------------------------------------
>>>     >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>
>>>     >>> For additional commands, e-mail:
>>>    java-dev-help@lucene.apache.org <mailto:
>>> java-dev-help@lucene.apache.org>
>>>
>>>     >>>
>>>     >>>
>>>     >>>
>>>     >>
>>>    ---------------------------------------------------------------------
>>>     >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>
>>>     >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>    <mailto:java-dev-help@lucene.apache.org>
>>>
>>>     >>
>>>     >>
>>>     >>
>>>     >
>>>     >
>>> ---------------------------------------------------------------------
>>>     > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>
>>>     > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>    <mailto:java-dev-help@lucene.apache.org>
>>>
>>>     >
>>>     >
>>>
>>>
>>>    --
>>>    - Mark
>>>
>>>    http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>>    ---------------------------------------------------------------------
>>>    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>
>>>    For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>    <mailto:java-dev-help@lucene.apache.org>
>>>
>>>
>>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>


-- 
- Mark

http://www.lucidimagination.com

Mime
View raw message