Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 44512 invoked from network); 20 Jan 2010 15:51:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 20 Jan 2010 15:51:07 -0000 Received: (qmail 95929 invoked by uid 500); 20 Jan 2010 15:51:05 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 95845 invoked by uid 500); 20 Jan 2010 15:51:05 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 95835 invoked by uid 99); 20 Jan 2010 15:51:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 15:51:04 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [192.112.254.53] (HELO smtp2.bedag.ch) (192.112.254.53) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 20 Jan 2010 15:50:56 +0000 Received: from localhost (localhost [127.0.0.1]) by smtp2.bedag.ch (Postfix) with ESMTP id E3038739A2 for ; Wed, 20 Jan 2010 16:50:35 +0100 (CET) X-Quarantine-ID: X-Virus-Scanned: Scanned Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: AW: NFS, Stale File Handle Problem and my thoughts.... Date: Wed, 20 Jan 2010 16:49:51 +0100 Message-ID: <1D3D7943573BD14FB024A9654EB43CCB02A4CC21@A99A-EMS-01VS2.ad.bedag.ch> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: NFS, Stale File Handle Problem and my thoughts.... Thread-Index: AcqZ4zkfL1zTY38+RyS20Qn6xgfCTAABHZ7w References: <1D3D7943573BD14FB024A9654EB43CCB02A4CC1E@A99A-EMS-01VS2.ad.bedag.ch> <9ac0c6aa1001200544w4ef5f26eq58b5d397c5ba1baa@mail.gmail.com> <1D3D7943573BD14FB024A9654EB43CCB02A4CC1F@A99A-EMS-01VS2.ad.bedag.ch> <9ac0c6aa1001200713n624674b3x529507ad3996a641@mail.gmail.com> From: "Sertic Mirko, Bedag" To: X-OriginalArrivalTime: 20 Jan 2010 15:50:30.0686 (UTC) FILETIME=[4B3753E0:01CA99E8] X-Virus-Checked: Checked by ClamAV on apache.org Mike Thank you so much for your feedback! Will the new IndexDeletionPolicy also be considered when segments are = merged? Does merging also affect the NFS problem? Should I use IndexReader.reOpen() or just create a new IndexReader? Thanks in advance Mirko -----Urspr=FCngliche Nachricht----- Von: Michael McCandless [mailto:lucene@mikemccandless.com]=20 Gesendet: Mittwoch, 20. Januar 2010 16:14 An: java-user@lucene.apache.org Betreff: Re: NFS, Stale File Handle Problem and my thoughts.... Right, it's only machine A that needs the deletion policy. All read-only machines just reopen on their schedule (or you can use some communication means a Shai describes to have lower latency reopen after the writer commits). Also realize that doing searching over NFS does not usually give very good performance... (though I've heard that mounting read-only can improve performance). Mike On Wed, Jan 20, 2010 at 9:05 AM, Sertic Mirko, Bedag wrote: > Hi Mike > > Thank you for your feedback! > > So I would need the following setup: > > a) Machine A with custom IndexDeletionPolicy and single IndexReader = instance > b) Machine B with custom IndexDeletionPolicy and single IndexReader = instance > c) Machine A and B periodically check if the index needs to be = reopened, at least at 12 o'clock > d) Machine A runs an Index update and optimization at 8 o'clock, using = the IndexDeletionPolicy. The IndexDeletionPolicy keeps track of the = files to be deleted. > e) On Machine A, the no longer needed files are taken from the = IndexDeletionPolicy, and deleted at 12:30. At this point the files to be = deleted should no longer be required by any IndexReader and can be = safely deleted. > > So the IndexDeletionPolicy should be a kind of Singleton, and in fact = would only be needed on Machine A, as only here index modifications are = made. Machine B has read only access. > > Would this be a valid setup? The only limitation is there is only ONE = IndexWriter box, and multiple IndexReader boxes. Based on our = requirements, this should fix very well. I really want to avoid some = kind of index replication between the boxes... > > Regards > Mirko > > > > -----Urspr=FCngliche Nachricht----- > Von: Michael McCandless [mailto:lucene@mikemccandless.com] > Gesendet: Mittwoch, 20. Januar 2010 14:45 > An: java-user@lucene.apache.org > Betreff: Re: NFS, Stale File Handle Problem and my thoughts.... > > Right, you just need to make a custom IndexDeletionPolicy. =A0NFS = makes > no effort to protect deletion of still-open files. > > A simple approach is one that only deletes a commit if it's more than > XXX minutes/hours old, such that XXX is set higher than the frequency > that IndexReaders are guaranteed to have reopened. > > The TestDeletionPolicy unit test in Lucene's sources has the > ExperiationTimeDeletionPolicy that you can use (but scrutinize!). > > Another alternative is to simply reopen the reader whenever you hit > Stale file handle (think of it as NFS's means of notifying you that > your reader is no longer current ;) ). =A0But, if reopen time is > potentially long for your app, it's no good to make queries pay that > cost and the deletion policy is better. > > Mike > > On Wed, Jan 20, 2010 at 8:29 AM, Sertic Mirko, Bedag > wrote: >> Hi@all >> >> >> >> We are using Lucene 2.4.1 on Debian Linux with 2 boxes. The index is >> stored on a common NFS share. Every box has a single IndexReader >> instance, and one Box has an IndexWriter instance, adding new = documents >> or deleting existing documents at a given point in time. After adding = or >> deleting the documents, a IndexWriter.optimize() is called. Every box >> checks periodically with IndexReader.isCurrent if the index needs to = be >> reopened. >> >> >> >> Now, we are encountering a "Stale file handle" error on box b after = the >> index was modified and optimized by box a. >> >> >> >> As far as i understand the problem with NFS is that box b tries to >> open/access a file that was deleted by box a on the NFS share. >> >> >> >> The question is now, when are files deleted? Does only the index >> optimization delete files, or can files be deleted just by adding or >> removing documents from an existing index? >> >> >> >> I now that there might be a better setup with Lucene and index >> replication, but this is an existing application and we cannot change >> the architecture now. So what would be the best solution? >> >> >> >> Can I just "change" the way Lucene deletes files? I think that just >> renaming no longer needed files would be good on NFS. After every >> IndexReader has reopened the index, the renamed files can be safely >> deleted, as they are definitely no longer needed. Where would be the >> hook point? I heard something about IndexDeletionPolicy.... >> >> >> >> Thanks in advance! >> >> >> >> Mirko >> >> >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-user-help@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-user-help@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org