lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: Nasty NIO behavior makes NIOFSDirectory silently close channel
Date Sun, 02 May 2010 22:13:03 GMT
Perfectly safe to use SimpleFSDirectory. When you measure the 
performance, are you measuring a lot of concurrent requests? That's 
where you should see the win.

- Mark

On 5/1/10 6:38 PM, John Wang wrote:
> We are seeing this issue as well in your production. (using Zoie on top
> of lucene 2.9.1)
> After some performance comparisons, we do NOT see performance gain with
> NIO, rather these nasty ClosedChannelExceptions.
>
> I think the performance gains ppl are seeing with 2.9.1 can be due to
> many different things. From what we seen, they are not related to
> NIOFSDirectory.
>
> Our solution is to avoid calling FSDirectory.open(), instead just call
> new SimpleFSDirectory(). Is this safe?
>
> -John
>
> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
> <mailto:markrmiller@gmail.com>> wrote:
>
>     Perhaps - one of the things they are supposed to be addressing is
>     extendability.
>
>     nio2 does have FileSystemProvider, which would actually allow you to
>     create a custom channel !
>
>     I have not dug in enough to know much more than that though.
>
>     *But*, another really interesting thing is that in Java 7,
>     FileDescriptors are ref counted ! (though users can't inc/dec).
>
>     But, FileInputStream and OutputStream have a new constructor that takes
>     a FileDescriptor.
>
>     So possibly, you could just make one that sits around to keep the
>     FileDescriptor valid, and get your channel off
>     FileInputStream/FileOutputStream?
>
>     And then if it goes down, make a new one using the FileDescriptor which
>     was not actually closed because there was a still a ref to it.
>
>     Possibly .... ;)
>
>     Michael McCandless wrote:
>      > Does anyone know if nio2 has improved this...?
>      >
>      > Mike
>      >
>      > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
>      > <jason.rutherglen@gmail.com <mailto:jason.rutherglen@gmail.com>>
>     wrote:
>      >
>      >> Defaulting NIOFSDir could account for some of the recent speed
>      >> improvements users have been reporting in Lucene 2.9.  So
>     removing it
>      >> as a default could reverse those and people could then report Lucene
>      >> 3.X has slowed...
>      >>
>      >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
>      >> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>
>     wrote:
>      >>
>      >>> Bummer.
>      >>>
>      >>> So the only viable workarounds are 1) don't use
>     Thread.interrupt (nor,
>      >>> things like Future.cancel, which in turn use Thread.interrupt) with
>      >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
>     app must
>      >>> make a deletion policy that keeps a commit alive if any reader is
>      >>> using it.  Or, 3) don't use NIOFSDir!
>      >>>
>      >>> Mike
>      >>>
>      >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
>      >>> <simon.willnauer@googlemail.com
>     <mailto:simon.willnauer@googlemail.com>> wrote:
>      >>>
>      >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
>      >>>> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>
>     wrote:
>      >>>>
>      >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
>     <uwe@thetaphi.de <mailto:uwe@thetaphi.de>> wrote:
>      >>>>>
>      >>>>>
>      >>>>>> So I checked the code of NIOFSIndexInput, my last comment
>     was not really correct:
>      >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
>     the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
>     until the file is closed (and also the channel).
>      >>>>>>
>      >>>>>> So it's really simple to fix in my opinion, just call
>     getChannel() again on this exception. Because the RAF should still
>     be open?
>      >>>>>>
>      >>>> Short answer:
>      >>>>  public final FileChannel getChannel() {
>      >>>>        synchronized (this) {
>      >>>>            if (channel == null)
>      >>>>                channel = FileChannelImpl.open(fd, true, rw, this);
>      >>>>            return channel;
>      >>>>        }
>      >>>>    }
>      >>>>
>      >>>> this is not gonna work I tried it before. The RandomAccessFile
>     buffers
>      >>>> the channel!!
>      >>>>
>      >>>> simon
>      >>>>
>      >>>>> I think we need a definitive answer on what happens to the
>     RAF when
>      >>>>> the FileChannel was closed by Thread.Interrupt.  Simon can
>     you test
>      >>>>> this?
>      >>>>>
>      >>>>> Mike
>      >>>>>
>      >>>>>
>     ---------------------------------------------------------------------
>      >>>>> To unsubscribe, e-mail:
>     java-dev-unsubscribe@lucene.apache.org
>     <mailto:java-dev-unsubscribe@lucene.apache.org>
>      >>>>> For additional commands, e-mail:
>     java-dev-help@lucene.apache.org <mailto:java-dev-help@lucene.apache.org>
>      >>>>>
>      >>>>>
>      >>>>>
>      >>>>
>     ---------------------------------------------------------------------
>      >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>     <mailto:java-dev-unsubscribe@lucene.apache.org>
>      >>>> For additional commands, e-mail:
>     java-dev-help@lucene.apache.org <mailto:java-dev-help@lucene.apache.org>
>      >>>>
>      >>>>
>      >>>>
>      >>>
>     ---------------------------------------------------------------------
>      >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>     <mailto:java-dev-unsubscribe@lucene.apache.org>
>      >>> For additional commands, e-mail:
>     java-dev-help@lucene.apache.org <mailto:java-dev-help@lucene.apache.org>
>      >>>
>      >>>
>      >>>
>      >>
>     ---------------------------------------------------------------------
>      >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>     <mailto:java-dev-unsubscribe@lucene.apache.org>
>      >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>     <mailto:java-dev-help@lucene.apache.org>
>      >>
>      >>
>      >>
>      >
>      > ---------------------------------------------------------------------
>      > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>     <mailto:java-dev-unsubscribe@lucene.apache.org>
>      > For additional commands, e-mail: java-dev-help@lucene.apache.org
>     <mailto:java-dev-help@lucene.apache.org>
>      >
>      >
>
>
>     --
>     - Mark
>
>     http://www.lucidimagination.com
>
>
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>     <mailto:java-dev-unsubscribe@lucene.apache.org>
>     For additional commands, e-mail: java-dev-help@lucene.apache.org
>     <mailto:java-dev-help@lucene.apache.org>
>
>


-- 
- Mark

http://www.lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message