Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of peterlkeegan@gmail.com
 designates 209.85.222.173 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=RTsSRWp5xYLs7bewY9IaBQpkafhZOeLfy57jk+bQcKwKTf9cjL+OrOru0/oDcaLnO+
         tRZDQwNzPoNt5GrtmKze7e77v+yeHhXwdvLiA2hCBEsnVJs4praCNzLko1UcC9mhlwyT
         fvrQ/1tFnDVzyEszxKAAX1gpEGe3CLoSb9M+E=
MIME-Version: 1.0
In-Reply-To: <9ac0c6aa0911070117j17895896m2b07c7fff03c771d@mail.gmail.com>
References: <e994873a0911060759p4095bfd3ud0c40238cf68833d@mail.gmail.com>
	 <9ac0c6aa0911060802l62484b3apd3c397dde66ae549@mail.gmail.com>
	 <e994873a0911060822m5a583aa9n216b38e94678c1df@mail.gmail.com>
	 <9ac0c6aa0911060833w36947b40oecf107dd7a15baaf@mail.gmail.com>
	 <e994873a0911060835t71f9532ao237fb186fad881e5@mail.gmail.com>
	 <9ac0c6aa0911060840j7194a61do8b25533c5a7d6064@mail.gmail.com>
	 <e994873a0911061146m60dc6038nc3611e17b6ed546f@mail.gmail.com>
	 <9ac0c6aa0911070117j17895896m2b07c7fff03c771d@mail.gmail.com>
Date: Sun, 8 Nov 2009 18:23:27 -0500
Message-ID: <e994873a0911081523m27627349x15f7070d1c4d2479@mail.gmail.com>
Subject: Re: 2 phase commit with external data
From: Peter Keegan <peterlkeegan@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=00504502ce8244680e0477e45d1a

--00504502ce8244680e0477e45d1a
Content-Type: text/plain; charset=ISO-8859-1

Here is some stand-alone code that reproduces the problem. There are 2
classes. jvm1 creates the index, jvm2 reads the index. The system console
input is used to synchronize the 4 steps.

jvm1:
--------------
import java.io.File;
import java.util.Scanner;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.store.SingleInstanceLockFactory;
import org.apache.lucene.search.DefaultSimilarity;
import org.apache.lucene.analysis.WhitespaceAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;


public class jvm1 {

    /**
     * @param args
     */
    public static void main(String[] args) {
        String indexPath;

        try {
            Scanner in = new Scanner(System.in);

            // create index
            indexPath = (args.length > 0) ? args[0] : "index";
            File idxFile  = new File(indexPath);
            idxFile.mkdirs();
            FSDirectory dir = FSDirectory.open(idxFile);
            SingleInstanceLockFactory lockFactory = new
SingleInstanceLockFactory();
            dir.setLockFactory(lockFactory);
            IndexWriter writer = new IndexWriter(dir, new
WhitespaceAnalyzer(), true, IndexWriter.MaxFieldLength.UNLIMITED);
            writer.setUseCompoundFile(false);
            writer.setSimilarity(new DefaultSimilarity());
            // Add some docs
            Document doc = new Document();
            doc.add(new Field("field", "aaa", Field.Store.YES,
Field.Index.ANALYZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
            for(int i=0;i<1000;i++) {
              writer.addDocument(doc);
            }
            writer.commit(); // flush to disk
            // Now wait for jvm2 to create reader
            System.out.println("Index created. Start jvm2 then hit 'Enter'
after jvm2 displays doc count");
            String input = in.nextLine();
            // Add some more docs, the prepare to commit
            for(int i=0;i<1000;i++) {
                  writer.addDocument(doc);
            }
            writer.prepareCommit();
            System.out.println("Index 'prepareCommit' called. Go to jvm2 and
hit 'Enter' (it should then call 'isCurrent')");
            System.out.println("Hit 'Enter' here to commit changes and close
index");
            input = in.nextLine();
            System.out.println("jvm1 about to commit/close index");
            writer.commit();
            writer.close();
            System.out.println("jvm1 done");

        } catch (Exception  e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

    }

}

jvm2:
-------------
import java.util.Scanner;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.store.FSDirectory;


public class jvm2 {

    /**
     * @param args
     */
    public static void main(String[] args) {
        String indexPath;
        try {
            Scanner in = new Scanner(System.in);
            indexPath = (args.length > 0) ? args[0] : "index";
            FSDirectory dir = FSDirectory.open(new java.io.File(indexPath));
            IndexReader reader  = IndexReader.open(dir, false);
            System.out.println("jvm2 running,  index doc count:
"+reader.numDocs());
            boolean isCurrent = reader.isCurrent();
            System.out.println("jvm2 isCurrent="+isCurrent);
            System.out.println("waiting for jvm1 to 'prepareCommit'. Hit
'Enter' when this happens");
            String input = in.nextLine();;
            isCurrent = reader.isCurrent();
            System.out.println("jvm2 isCurrent="+isCurrent);
            reader.close();
            System.out.println("done");

        } catch (Exception  e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

    }

}

Peter


On Sat, Nov 7, 2009 at 4:17 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> Hmm... for step 4 you should have gotten "true" back from isCurrent.
> You're sure there were no intervening calls to IndexWriter.commit?
> Are you using Lucene 2.9?   If not, you have to make sure autoCommit
> is false when opening the IndexWriter.
>
> Mike
>
> On Fri, Nov 6, 2009 at 2:46 PM, Peter Keegan <peterlkeegan@gmail.com>
> wrote:
> > Here's a new scenario:
> >
> > 1. JVM 1 creates IndexWriter, version 1
> > 2. JVM 2 creates IndexReader, version 1
> > 3. JVM 1 IndexWriter calls prepareCommit()
> > 4. JVM 2 IndexReader.isCurrent() returns false
> >
> > In step 4, I expected 'isCurrent' to return true until the IndexWriter
> had
> > committed in JVM 1. Is this the correct behavior?
> >
> > Peter
> >
> >
> > On Fri, Nov 6, 2009 at 11:40 AM, Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> >> It will always return a reader reflecting every change done with that
> >> writer (plus, the index as it was when the writer was opened) before
> >> getReader was called.
> >>
> >> It's unaffected by the call to prepareCommit.
> >>
> >> Mike
> >>
> >> On Fri, Nov 6, 2009 at 11:35 AM, Peter Keegan <peterlkeegan@gmail.com>
> >> wrote:
> >> > Which version of the index will IndexWriter.getReader() return if
> there
> >> have
> >> > been updates, but no call to 'prepareCommit'?
> >> >
> >> >
> >> > On Fri, Nov 6, 2009 at 11:33 AM, Michael McCandless <
> >> > lucene@mikemccandless.com> wrote:
> >> >
> >> >> On Fri, Nov 6, 2009 at 11:22 AM, Peter Keegan <
> peterlkeegan@gmail.com>
> >> >> wrote:
> >> >> >>Can you use IndexWriter.getReader() to get the reader for step 2
> >> >> > Yes - perfect! I didn't think that would be different than
> refreshing
> >> or
> >> >> > recreating an IndexReader.
> >> >>
> >> >> Great!
> >> >>
> >> >> getReader() searches the full index, plus uncommitted changes.
> >> >>
> >> >> > I don't need to keep the old commit alive. The goal is to keep the
> >> >> external
> >> >> > file in synch with the index, so a separate searcher process will
> see
> >> >> > consistent data. By postponing both commits, the window where they
> are
> >> >> out
> >> >> > of synch is very small (2 file renames). I record the Lucene index
> >> >> version
> >> >> > in the external file for checking synchcronization.
> >> >>
> >> >> OK.
> >> >>
> >> >> Mike
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >>
> >> >>
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--00504502ce8244680e0477e45d1a--