lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Date Mon, 03 May 2010 09:47:42 GMT
This is strange -- it's completely opposite of what others have seen!

Other have seen sizable concurrency gains when switching from
SimpleFSDir -> NIOFSDir, except on Windows where Sun's long standing
JVM bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734)
prevents concurrency gains.

There must be something different about the testing.

John are you testing pure searching?  Are you reopening readers as
well?  How much HW concurrency do your machines have?

Mike

On Sun, May 2, 2010 at 11:53 PM, John Wang <john.wang@gmail.com> wrote:
> Solaris and MacOs
>
> On Sun, May 2, 2010 at 4:47 PM, Mark Miller <markrmiller@gmail.com> wrote:
>>
>> Interesting ... whats the OS?
>>
>> On Sun, May 2, 2010 at 7:12 PM, John Wang <john.wang@gmail.com> wrote:
>>>
>>> 12 concurrent search threads in a stress environment. (with about 5M doc
>>> index)
>>> -John
>>>
>>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <markrmiller@gmail.com>
>>> wrote:
>>>>
>>>> Perfectly safe to use SimpleFSDirectory. When you measure the
>>>> performance, are you measuring a lot of concurrent requests? That's where
>>>> you should see the win.
>>>>
>>>> - Mark
>>>>
>>>> On 5/1/10 6:38 PM, John Wang wrote:
>>>>>
>>>>> We are seeing this issue as well in your production. (using Zoie on top
>>>>> of lucene 2.9.1)
>>>>> After some performance comparisons, we do NOT see performance gain with
>>>>> NIO, rather these nasty ClosedChannelExceptions.
>>>>>
>>>>> I think the performance gains ppl are seeing with 2.9.1 can be due to
>>>>> many different things. From what we seen, they are not related to
>>>>> NIOFSDirectory.
>>>>>
>>>>> Our solution is to avoid calling FSDirectory.open(), instead just call
>>>>> new SimpleFSDirectory(). Is this safe?
>>>>>
>>>>> -John
>>>>>
>>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
>>>>> <mailto:markrmiller@gmail.com>> wrote:
>>>>>
>>>>>    Perhaps - one of the things they are supposed to be addressing is
>>>>>    extendability.
>>>>>
>>>>>    nio2 does have FileSystemProvider, which would actually allow you
to
>>>>>    create a custom channel !
>>>>>
>>>>>    I have not dug in enough to know much more than that though.
>>>>>
>>>>>    *But*, another really interesting thing is that in Java 7,
>>>>>    FileDescriptors are ref counted ! (though users can't inc/dec).
>>>>>
>>>>>    But, FileInputStream and OutputStream have a new constructor that
>>>>> takes
>>>>>    a FileDescriptor.
>>>>>
>>>>>    So possibly, you could just make one that sits around to keep the
>>>>>    FileDescriptor valid, and get your channel off
>>>>>    FileInputStream/FileOutputStream?
>>>>>
>>>>>    And then if it goes down, make a new one using the FileDescriptor
>>>>> which
>>>>>    was not actually closed because there was a still a ref to it.
>>>>>
>>>>>    Possibly .... ;)
>>>>>
>>>>>    Michael McCandless wrote:
>>>>>     > Does anyone know if nio2 has improved this...?
>>>>>     >
>>>>>     > Mike
>>>>>     >
>>>>>     > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>>>>>     > <jason.rutherglen@gmail.com <mailto:jason.rutherglen@gmail.com>>
>>>>>    wrote:
>>>>>     >
>>>>>     >> Defaulting NIOFSDir could account for some of the recent
speed
>>>>>     >> improvements users have been reporting in Lucene 2.9.
 So
>>>>>    removing it
>>>>>     >> as a default could reverse those and people could then
report
>>>>> Lucene
>>>>>     >> 3.X has slowed...
>>>>>     >>
>>>>>     >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>>>>>     >> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>
>>>>>    wrote:
>>>>>     >>
>>>>>     >>> Bummer.
>>>>>     >>>
>>>>>     >>> So the only viable workarounds are 1) don't use
>>>>>    Thread.interrupt (nor,
>>>>>     >>> things like Future.cancel, which in turn use Thread.interrupt)
>>>>> with
>>>>>     >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel
AND the
>>>>>    app must
>>>>>     >>> make a deletion policy that keeps a commit alive if
any reader
>>>>> is
>>>>>     >>> using it.  Or, 3) don't use NIOFSDir!
>>>>>     >>>
>>>>>     >>> Mike
>>>>>     >>>
>>>>>     >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>>>>>     >>> <simon.willnauer@googlemail.com
>>>>>    <mailto:simon.willnauer@googlemail.com>> wrote:
>>>>>     >>>
>>>>>     >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>>>>>     >>>> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>
>>>>>    wrote:
>>>>>     >>>>
>>>>>     >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>>>>>    <uwe@thetaphi.de <mailto:uwe@thetaphi.de>> wrote:
>>>>>     >>>>>
>>>>>     >>>>>
>>>>>     >>>>>> So I checked the code of NIOFSIndexInput,
my last comment
>>>>>    was not really correct:
>>>>>     >>>>>> NIOFSIndexInput extends SimpleFSIndexInput
and that opens
>>>>>    the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>>>>>    until the file is closed (and also the channel).
>>>>>     >>>>>>
>>>>>     >>>>>> So it's really simple to fix in my opinion,
just call
>>>>>    getChannel() again on this exception. Because the RAF should still
>>>>>    be open?
>>>>>     >>>>>>
>>>>>     >>>> Short answer:
>>>>>     >>>>  public final FileChannel getChannel() {
>>>>>     >>>>        synchronized (this) {
>>>>>     >>>>            if (channel == null)
>>>>>     >>>>                channel = FileChannelImpl.open(fd,
true, rw,
>>>>> this);
>>>>>     >>>>            return channel;
>>>>>     >>>>        }
>>>>>     >>>>    }
>>>>>     >>>>
>>>>>     >>>> this is not gonna work I tried it before. The
RandomAccessFile
>>>>>    buffers
>>>>>     >>>> the channel!!
>>>>>     >>>>
>>>>>     >>>> simon
>>>>>     >>>>
>>>>>     >>>>> I think we need a definitive answer on what
happens to the
>>>>>    RAF when
>>>>>     >>>>> the FileChannel was closed by Thread.Interrupt.
 Simon can
>>>>>    you test
>>>>>     >>>>> this?
>>>>>     >>>>>
>>>>>     >>>>> Mike
>>>>>     >>>>>
>>>>>     >>>>>
>>>>>
>>>>>  ---------------------------------------------------------------------
>>>>>     >>>>> To unsubscribe, e-mail:
>>>>>    java-dev-unsubscribe@lucene.apache.org
>>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>>     >>>>> For additional commands, e-mail:
>>>>>    java-dev-help@lucene.apache.org
>>>>> <mailto:java-dev-help@lucene.apache.org>
>>>>>     >>>>>
>>>>>     >>>>>
>>>>>     >>>>>
>>>>>     >>>>
>>>>>
>>>>>  ---------------------------------------------------------------------
>>>>>     >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>>     >>>> For additional commands, e-mail:
>>>>>    java-dev-help@lucene.apache.org
>>>>> <mailto:java-dev-help@lucene.apache.org>
>>>>>     >>>>
>>>>>     >>>>
>>>>>     >>>>
>>>>>     >>>
>>>>>
>>>>>  ---------------------------------------------------------------------
>>>>>     >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>>     >>> For additional commands, e-mail:
>>>>>    java-dev-help@lucene.apache.org
>>>>> <mailto:java-dev-help@lucene.apache.org>
>>>>>     >>>
>>>>>     >>>
>>>>>     >>>
>>>>>     >>
>>>>>
>>>>>  ---------------------------------------------------------------------
>>>>>     >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>>     >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>    <mailto:java-dev-help@lucene.apache.org>
>>>>>     >>
>>>>>     >>
>>>>>     >>
>>>>>     >
>>>>>     >
>>>>> ---------------------------------------------------------------------
>>>>>     > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>>     > For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>    <mailto:java-dev-help@lucene.apache.org>
>>>>>     >
>>>>>     >
>>>>>
>>>>>
>>>>>    --
>>>>>    - Mark
>>>>>
>>>>>    http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>  ---------------------------------------------------------------------
>>>>>    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>    <mailto:java-dev-unsubscribe@lucene.apache.org>
>>>>>    For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>    <mailto:java-dev-help@lucene.apache.org>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> - Mark
>>>>
>>>> http://www.lucidimagination.com
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
>>
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message