lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Keegan <peterlkee...@gmail.com>
Subject Re: 2 phase commit with external data
Date Sun, 08 Nov 2009 23:23:27 GMT
Here is some stand-alone code that reproduces the problem. There are 2
classes. jvm1 creates the index, jvm2 reads the index. The system console
input is used to synchronize the 4 steps.

jvm1:
--------------
import java.io.File;
import java.util.Scanner;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.store.SingleInstanceLockFactory;
import org.apache.lucene.search.DefaultSimilarity;
import org.apache.lucene.analysis.WhitespaceAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;



public class jvm1 {

    /**
     * @param args
     */
    public static void main(String[] args) {
        String indexPath;

        try {
            Scanner in = new Scanner(System.in);

            // create index
            indexPath = (args.length > 0) ? args[0] : "index";
            File idxFile  = new File(indexPath);
            idxFile.mkdirs();
            FSDirectory dir = FSDirectory.open(idxFile);
            SingleInstanceLockFactory lockFactory = new
SingleInstanceLockFactory();
            dir.setLockFactory(lockFactory);
            IndexWriter writer = new IndexWriter(dir, new
WhitespaceAnalyzer(), true, IndexWriter.MaxFieldLength.UNLIMITED);
            writer.setUseCompoundFile(false);
            writer.setSimilarity(new DefaultSimilarity());
            // Add some docs
            Document doc = new Document();
            doc.add(new Field("field", "aaa", Field.Store.YES,
Field.Index.ANALYZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
            for(int i=0;i<1000;i++) {
              writer.addDocument(doc);
            }
            writer.commit(); // flush to disk
            // Now wait for jvm2 to create reader
            System.out.println("Index created. Start jvm2 then hit 'Enter'
after jvm2 displays doc count");
            String input = in.nextLine();
            // Add some more docs, the prepare to commit
            for(int i=0;i<1000;i++) {
                  writer.addDocument(doc);
            }
            writer.prepareCommit();
            System.out.println("Index 'prepareCommit' called. Go to jvm2 and
hit 'Enter' (it should then call 'isCurrent')");
            System.out.println("Hit 'Enter' here to commit changes and close
index");
            input = in.nextLine();
            System.out.println("jvm1 about to commit/close index");
            writer.commit();
            writer.close();
            System.out.println("jvm1 done");

        } catch (Exception  e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

    }

}

jvm2:
-------------
import java.util.Scanner;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.store.FSDirectory;


public class jvm2 {

    /**
     * @param args
     */
    public static void main(String[] args) {
        String indexPath;
        try {
            Scanner in = new Scanner(System.in);
            indexPath = (args.length > 0) ? args[0] : "index";
            FSDirectory dir = FSDirectory.open(new java.io.File(indexPath));
            IndexReader reader  = IndexReader.open(dir, false);
            System.out.println("jvm2 running,  index doc count:
"+reader.numDocs());
            boolean isCurrent = reader.isCurrent();
            System.out.println("jvm2 isCurrent="+isCurrent);
            System.out.println("waiting for jvm1 to 'prepareCommit'. Hit
'Enter' when this happens");
            String input = in.nextLine();;
            isCurrent = reader.isCurrent();
            System.out.println("jvm2 isCurrent="+isCurrent);
            reader.close();
            System.out.println("done");

        } catch (Exception  e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

    }

}

Peter



On Sat, Nov 7, 2009 at 4:17 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> Hmm... for step 4 you should have gotten "true" back from isCurrent.
> You're sure there were no intervening calls to IndexWriter.commit?
> Are you using Lucene 2.9?   If not, you have to make sure autoCommit
> is false when opening the IndexWriter.
>
> Mike
>
> On Fri, Nov 6, 2009 at 2:46 PM, Peter Keegan <peterlkeegan@gmail.com>
> wrote:
> > Here's a new scenario:
> >
> > 1. JVM 1 creates IndexWriter, version 1
> > 2. JVM 2 creates IndexReader, version 1
> > 3. JVM 1 IndexWriter calls prepareCommit()
> > 4. JVM 2 IndexReader.isCurrent() returns false
> >
> > In step 4, I expected 'isCurrent' to return true until the IndexWriter
> had
> > committed in JVM 1. Is this the correct behavior?
> >
> > Peter
> >
> >
> > On Fri, Nov 6, 2009 at 11:40 AM, Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> >> It will always return a reader reflecting every change done with that
> >> writer (plus, the index as it was when the writer was opened) before
> >> getReader was called.
> >>
> >> It's unaffected by the call to prepareCommit.
> >>
> >> Mike
> >>
> >> On Fri, Nov 6, 2009 at 11:35 AM, Peter Keegan <peterlkeegan@gmail.com>
> >> wrote:
> >> > Which version of the index will IndexWriter.getReader() return if
> there
> >> have
> >> > been updates, but no call to 'prepareCommit'?
> >> >
> >> >
> >> > On Fri, Nov 6, 2009 at 11:33 AM, Michael McCandless <
> >> > lucene@mikemccandless.com> wrote:
> >> >
> >> >> On Fri, Nov 6, 2009 at 11:22 AM, Peter Keegan <
> peterlkeegan@gmail.com>
> >> >> wrote:
> >> >> >>Can you use IndexWriter.getReader() to get the reader for step
2
> >> >> > Yes - perfect! I didn't think that would be different than
> refreshing
> >> or
> >> >> > recreating an IndexReader.
> >> >>
> >> >> Great!
> >> >>
> >> >> getReader() searches the full index, plus uncommitted changes.
> >> >>
> >> >> > I don't need to keep the old commit alive. The goal is to keep
the
> >> >> external
> >> >> > file in synch with the index, so a separate searcher process will
> see
> >> >> > consistent data. By postponing both commits, the window where
they
> are
> >> >> out
> >> >> > of synch is very small (2 file renames). I record the Lucene index
> >> >> version
> >> >> > in the external file for checking synchcronization.
> >> >>
> >> >> OK.
> >> >>
> >> >> Mike
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >>
> >> >>
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message