From dev-return-50141-apmail-lucene-dev-archive=lucene.apache.org@lucene.apache.org Sun May 02 22:13:36 2010 Return-Path: Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: (qmail 99945 invoked from network); 2 May 2010 22:13:35 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 2 May 2010 22:13:35 -0000 Received: (qmail 33863 invoked by uid 500); 2 May 2010 22:13:34 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 33794 invoked by uid 500); 2 May 2010 22:13:34 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 33787 invoked by uid 99); 2 May 2010 22:13:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 May 2010 22:13:34 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of markrmiller@gmail.com designates 209.85.221.173 as permitted sender) Received: from [209.85.221.173] (HELO mail-qy0-f173.google.com) (209.85.221.173) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 May 2010 22:13:28 +0000 Received: by qyk4 with SMTP id 4so3037783qyk.17 for ; Sun, 02 May 2010 15:13:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:references:in-reply-to :content-type:content-transfer-encoding; bh=r+S83P621FuDtTkEfKsByoqMcFLUfv2qw1EwU7CTxFE=; b=USXc/KH/LX9Kbjy0kM2yRaGh5rE/YqWToWFqZqZspJPoFTwAhPS7T06Dih0TtuNkxh bFfuBu8rr9UXwSJLYrsLOkzA5X/VdqV+kvhCO9LwyZy1Uy7eY+kYjvaRKPCQV1s5Wjx3 3JD1X+9p5hBAQBfbox/vQtXm2GaPRL9BV/LxY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; b=ZthuPleUhBWuoiqPdyp4B7bs3qiAxLSyRO+cTZh2YvnVEf+PONXJF2ZJ6wpTm76wqZ IGci9158x76opEsvgnRW+AUoAEIYF3e2X3XcNlJ/ks0Q87X4fuP5+0BtrpBhogT2dfgD D54ZMVLlLF5nbtmd52ImGHz6+FRQbW+g9VuZQ= Received: by 10.224.123.18 with SMTP id n18mr2650339qar.244.1272838385791; Sun, 02 May 2010 15:13:05 -0700 (PDT) Received: from Mark-Millers-MacBook-Pro.local (ool-44c639d9.dyn.optonline.net [68.198.57.217]) by mx.google.com with ESMTPS id 5sm7911495qwh.55.2010.05.02.15.13.03 (version=SSLv3 cipher=RC4-MD5); Sun, 02 May 2010 15:13:04 -0700 (PDT) Message-ID: <4BDDF8EF.8040700@gmail.com> Date: Sun, 02 May 2010 18:13:03 -0400 From: Mark Miller User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: dev@lucene.apache.org Subject: Re: Nasty NIO behavior makes NIOFSDirectory silently close channel References: <000101caa00c$463a7b90$d2af72b0$@de> <000201caa00d$138a4c10$3a9ee430$@de> <000901caa00e$6c538680$44fa9380$@de> <9ac0c6aa1001280343m3449a542kc6b2dea5d1c6aa23@mail.gmail.com> <9ac0c6aa1001280524t4ce21010ta6f00a94a44a9864@mail.gmail.com> <85d3c3b61001291100n4fe0fa33v2da129d94d5d1e1e@mail.gmail.com> <9ac0c6aa1001291103m4b8b410q309df938fbb173c2@mail.gmail.com> <4B6337BE.1080701@gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Perfectly safe to use SimpleFSDirectory. When you measure the performance, are you measuring a lot of concurrent requests? That's where you should see the win. - Mark On 5/1/10 6:38 PM, John Wang wrote: > We are seeing this issue as well in your production. (using Zoie on top > of lucene 2.9.1) > After some performance comparisons, we do NOT see performance gain with > NIO, rather these nasty ClosedChannelExceptions. > > I think the performance gains ppl are seeing with 2.9.1 can be due to > many different things. From what we seen, they are not related to > NIOFSDirectory. > > Our solution is to avoid calling FSDirectory.open(), instead just call > new SimpleFSDirectory(). Is this safe? > > -John > > On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller > wrote: > > Perhaps - one of the things they are supposed to be addressing is > extendability. > > nio2 does have FileSystemProvider, which would actually allow you to > create a custom channel ! > > I have not dug in enough to know much more than that though. > > *But*, another really interesting thing is that in Java 7, > FileDescriptors are ref counted ! (though users can't inc/dec). > > But, FileInputStream and OutputStream have a new constructor that takes > a FileDescriptor. > > So possibly, you could just make one that sits around to keep the > FileDescriptor valid, and get your channel off > FileInputStream/FileOutputStream? > > And then if it goes down, make a new one using the FileDescriptor which > was not actually closed because there was a still a ref to it. > > Possibly .... ;) > > Michael McCandless wrote: > > Does anyone know if nio2 has improved this...? > > > > Mike > > > > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen > > > > wrote: > > > >> Defaulting NIOFSDir could account for some of the recent speed > >> improvements users have been reporting in Lucene 2.9. So > removing it > >> as a default could reverse those and people could then report Lucene > >> 3.X has slowed... > >> > >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless > >> > > wrote: > >> > >>> Bummer. > >>> > >>> So the only viable workarounds are 1) don't use > Thread.interrupt (nor, > >>> things like Future.cancel, which in turn use Thread.interrupt) with > >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the > app must > >>> make a deletion policy that keeps a commit alive if any reader is > >>> using it. Or, 3) don't use NIOFSDir! > >>> > >>> Mike > >>> > >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer > >>> > wrote: > >>> > >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless > >>>> > > wrote: > >>>> > >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler > > wrote: > >>>>> > >>>>> > >>>>>> So I checked the code of NIOFSIndexInput, my last comment > was not really correct: > >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens > the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open > until the file is closed (and also the channel). > >>>>>> > >>>>>> So it's really simple to fix in my opinion, just call > getChannel() again on this exception. Because the RAF should still > be open? > >>>>>> > >>>> Short answer: > >>>> public final FileChannel getChannel() { > >>>> synchronized (this) { > >>>> if (channel == null) > >>>> channel = FileChannelImpl.open(fd, true, rw, this); > >>>> return channel; > >>>> } > >>>> } > >>>> > >>>> this is not gonna work I tried it before. The RandomAccessFile > buffers > >>>> the channel!! > >>>> > >>>> simon > >>>> > >>>>> I think we need a definitive answer on what happens to the > RAF when > >>>>> the FileChannel was closed by Thread.Interrupt. Simon can > you test > >>>>> this? > >>>>> > >>>>> Mike > >>>>> > >>>>> > --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: > java-dev-unsubscribe@lucene.apache.org > > >>>>> For additional commands, e-mail: > java-dev-help@lucene.apache.org > >>>>> > >>>>> > >>>>> > >>>> > --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > > >>>> For additional commands, e-mail: > java-dev-help@lucene.apache.org > >>>> > >>>> > >>>> > >>> > --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > > >>> For additional commands, e-mail: > java-dev-help@lucene.apache.org > >>> > >>> > >>> > >> > --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > > >> For additional commands, e-mail: java-dev-help@lucene.apache.org > > >> > >> > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > > > For additional commands, e-mail: java-dev-help@lucene.apache.org > > > > > > > > -- > - Mark > > http://www.lucidimagination.com > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > > For additional commands, e-mail: java-dev-help@lucene.apache.org > > > -- - Mark http://www.lucidimagination.com --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org