lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dyzc2010" <zhoucheng2...@gmail.com>
Subject Re: How to use RAMDirectory more efficiently
Date Tue, 03 Jan 2012 21:20:10 GMT
Charlie,


the code you provided will double the size of the index on FS every time when saving occurs.


Can we avoid duplicating the index but synchronizing the changed records?
 
------------------ Original ------------------
From:  "dyzc2010  "<1393975679@qq.com>;
Date:  Mon, Jan 2, 2012 01:30 PM
To:  "java-user"<java-user@lucene.apache.org>; 

Subject:  Re: How to use RAMDirectory more efficiently

 
will the solution provided by Charlie Hubbard do the work?


I am not sure I understand the "writer.addIndexes( ram );" part.


What if there are inconsistent data between FS and RAM? What if in a multi-threading environment
one thread try to synchronize the two while the other thread keeps writing to FS?


In all, my question is that I want to build a RAM-based index based upon a FS based one. After
all the modifications to the RAM-based, I want to write it back (maybe the word "synchronize"
is more accurate) to the FS.


How can I do the write back work?


I found some example that uses "Directory.write". But it doesn't look right to me.
 
 
------------------ Original ------------------
From:  "Greg Steffensen"<greg.steffensen@gmail.com>;
Date:  Mon, Jan 2, 2012 12:54 PM
To:  "java-user"<java-user@lucene.apache.org>; 

Subject:  Re: How to use RAMDirectory more efficiently

 
The two directories won't be synchronized.  As the RAMDirectory JavaDocs
say:

Note that the resulting RAMDirectory instance is fully independent from the
original Directory (it is a complete copy). Any subsequent changes to the
originalDirectory will not be visible in the RAMDirectory instance.

So writes using that IndexWriter will only appear in ramDir.

On Sun, Jan 1, 2012 at 11:21 PM, Cheng <zhoucheng2008@gmail.com> wrote:

> what about my code as follow:
>
> FSDirectory indexDir = new NIOFSDirectory(new File("c:/index_folder"));
> Directory ramDir = new RAMDirectory(indexDir);
> IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_35,
> new StandardAnalyzer(Version.LUCENE_35));
> IndexWriter iw = new IndexWriter(ramDir, iwc);
>
> I associate the FSDirectory and RAMDirectory as the very beginning. Will
> the two be synchronized when the writer is committed or close?
>
> Thanks
>
>
>
> On Sun, Jan 1, 2012 at 2:56 PM, Charlie Hubbard
> <charlie.hubbard@gmail.com>wrote:
>
> > You can always index into RAMDirectory for speed then synchronize those
> > changes to the disk by adding the RAMDirectory to a FSDirectory at some
> > point.  Here is a simple example of how to do that:
> >
> >    public void save( RAMDirectory ram, File dir ) {
> >       FSDirectory fs = FSDirectory.open( dir );
> >       IndexWriter writer = new IndexWriter( fs, ... );
> >       try {
> >            writer.addIndexes( ram );
> >       } finally {
> >             writer.close();
> >       }
> >   }
> >
> >
> >
> http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/index/IndexWriter.html#addIndexes(org.apache.lucene.store.Directory
> > ..
> > .)
> >
> > On Sat, Dec 31, 2011 at 3:37 PM, Cheng <zhoucheng2008@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Suppose that we have a huge amount of indices on hard drives but
> working
> > in
> > > RAMDirectory is a must, how can we decide which part of the indices to
> be
> > > loaded into RAM, how to modify the indices, and when and how to
> > synchronize
> > > the indices with those on hard drives?
> > >
> > > Any thoughts?
> > >
> > > Thanks!
> > >
> >
>
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message