Interesting ... whats the OS?

On Sun, May 2, 2010 at 7:12 PM, John Wang <john.wang@gmail.com> wrote:
12 concurrent search threads in a stress environment. (with about 5M doc index)

-John

On Sun, May 2, 2010 at 3:13 PM, Mark Miller <markrmiller@gmail.com> wrote:
Perfectly safe to use SimpleFSDirectory. When you measure the performance, are you measuring a lot of concurrent requests? That's where you should see the win.

- Mark


On 5/1/10 6:38 PM, John Wang wrote:
We are seeing this issue as well in your production. (using Zoie on top
of lucene 2.9.1)
After some performance comparisons, we do NOT see performance gain with
NIO, rather these nasty ClosedChannelExceptions.

I think the performance gains ppl are seeing with 2.9.1 can be due to
many different things. From what we seen, they are not related to
NIOFSDirectory.

Our solution is to avoid calling FSDirectory.open(), instead just call
new SimpleFSDirectory(). Is this safe?

-John

On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmiller@gmail.com
<mailto:markrmiller@gmail.com>> wrote:

   Perhaps - one of the things they are supposed to be addressing is
   extendability.

   nio2 does have FileSystemProvider, which would actually allow you to
   create a custom channel !

   I have not dug in enough to know much more than that though.

   *But*, another really interesting thing is that in Java 7,
   FileDescriptors are ref counted ! (though users can't inc/dec).

   But, FileInputStream and OutputStream have a new constructor that takes
   a FileDescriptor.

   So possibly, you could just make one that sits around to keep the
   FileDescriptor valid, and get your channel off
   FileInputStream/FileOutputStream?

   And then if it goes down, make a new one using the FileDescriptor which
   was not actually closed because there was a still a ref to it.

   Possibly .... ;)

   Michael McCandless wrote:
    > Does anyone know if nio2 has improved this...?
    >
    > Mike
    >
    > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen
    > <jason.rutherglen@gmail.com <mailto:jason.rutherglen@gmail.com>>

   wrote:
    >
    >> Defaulting NIOFSDir could account for some of the recent speed
    >> improvements users have been reporting in Lucene 2.9.  So
   removing it
    >> as a default could reverse those and people could then report Lucene
    >> 3.X has slowed...
    >>
    >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless
    >> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>

   wrote:
    >>
    >>> Bummer.
    >>>
    >>> So the only viable workarounds are 1) don't use
   Thread.interrupt (nor,
    >>> things like Future.cancel, which in turn use Thread.interrupt) with
    >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the
   app must
    >>> make a deletion policy that keeps a commit alive if any reader is
    >>> using it.  Or, 3) don't use NIOFSDir!
    >>>
    >>> Mike
    >>>
    >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer
    >>> <simon.willnauer@googlemail.com
   <mailto:simon.willnauer@googlemail.com>> wrote:
    >>>
    >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless
    >>>> <lucene@mikemccandless.com <mailto:lucene@mikemccandless.com>>

   wrote:
    >>>>
    >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler
   <uwe@thetaphi.de <mailto:uwe@thetaphi.de>> wrote:
    >>>>>
    >>>>>
    >>>>>> So I checked the code of NIOFSIndexInput, my last comment
   was not really correct:
    >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens
   the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open
   until the file is closed (and also the channel).
    >>>>>>
    >>>>>> So it's really simple to fix in my opinion, just call
   getChannel() again on this exception. Because the RAF should still
   be open?
    >>>>>>
    >>>> Short answer:
    >>>>  public final FileChannel getChannel() {
    >>>>        synchronized (this) {
    >>>>            if (channel == null)
    >>>>                channel = FileChannelImpl.open(fd, true, rw, this);
    >>>>            return channel;
    >>>>        }
    >>>>    }
    >>>>
    >>>> this is not gonna work I tried it before. The RandomAccessFile
   buffers
    >>>> the channel!!
    >>>>
    >>>> simon
    >>>>
    >>>>> I think we need a definitive answer on what happens to the
   RAF when
    >>>>> the FileChannel was closed by Thread.Interrupt.  Simon can
   you test
    >>>>> this?
    >>>>>
    >>>>> Mike
    >>>>>
    >>>>>
   ---------------------------------------------------------------------
    >>>>> To unsubscribe, e-mail:
   java-dev-unsubscribe@lucene.apache.org
   <mailto:java-dev-unsubscribe@lucene.apache.org>

    >>>>> For additional commands, e-mail:
   java-dev-help@lucene.apache.org <mailto:java-dev-help@lucene.apache.org>

    >>>>>
    >>>>>
    >>>>>
    >>>>
   ---------------------------------------------------------------------
    >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
   <mailto:java-dev-unsubscribe@lucene.apache.org>

    >>>> For additional commands, e-mail:
   java-dev-help@lucene.apache.org <mailto:java-dev-help@lucene.apache.org>

    >>>>
    >>>>
    >>>>
    >>>
   ---------------------------------------------------------------------
    >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
   <mailto:java-dev-unsubscribe@lucene.apache.org>

    >>> For additional commands, e-mail:
   java-dev-help@lucene.apache.org <mailto:java-dev-help@lucene.apache.org>

    >>>
    >>>
    >>>
    >>
   ---------------------------------------------------------------------
    >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
   <mailto:java-dev-unsubscribe@lucene.apache.org>

    >> For additional commands, e-mail: java-dev-help@lucene.apache.org
   <mailto:java-dev-help@lucene.apache.org>

    >>
    >>
    >>
    >
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
   <mailto:java-dev-unsubscribe@lucene.apache.org>

    > For additional commands, e-mail: java-dev-help@lucene.apache.org
   <mailto:java-dev-help@lucene.apache.org>

    >
    >


   --
   - Mark

   http://www.lucidimagination.com




   ---------------------------------------------------------------------
   To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
   <mailto:java-dev-unsubscribe@lucene.apache.org>

   For additional commands, e-mail: java-dev-help@lucene.apache.org
   <mailto:java-dev-help@lucene.apache.org>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org





--
- Mark

http://www.lucidimagination.com