lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Pattern for maintaining FSDirectory copy of RAMDirectory
Date Mon, 16 Feb 2009 16:14:49 GMT
WARNING: I haven't had occasion to use the Directory.copy
method, so I'm talking a bit theoretically here.....


I guess my main issue with your scheme is the usual
abnormal termination issues and how you can be absolutely
sure boust what's in your FSDir. So I guess what I'd
do is create some kind of marker for whether your app was
gracefully shut down. Then, when gracefully shutting down,
use Directory.copy to copy FROM your RAMDir TO your
FSDir.

Upon startup, you can check your marker and, if all is well,
read your FSDirectory into your RAMDir, and away you go.
If you have any reason to doubt, you can reconstruct your
RAMDir from the DB.

Then you wouldn't have to worry in the least about making
parallel, transactional updates, you'd just use your RAMDir
and go from there.

Of course you could periodically do this copy if you didn't
regularly shut down.....

I leave the construction of the marker as an exercise for
the reader <G>...

Best
Erick

On Mon, Feb 16, 2009 at 10:49 AM, Joel Halbert <joel@su3analytics.com>wrote:

> Hi,
>
> I have a RAMDirectory based index. The document source for the index is
> a database table, where content to be indexed is stored alongside a
> status (pending_index, indexed, pending_delete, deleted). Each time the
> application is started, and periodically thereafter, all documents from
> the database that are "pending_index" or "indexed" are loaded and added
> to the index. Once this is done their status is updated to
> "indexed" (and this process repeats itself as required).
>
> I would now like to keep a FSDir copy of the index, to speedup the
> application start time. I am proposing to simply update both the FSDir
> and RAMDir in tandem i.e. whenever a document is to be added to the
> index, both Dirs are updated in one "transaction" (and the database
> record is then updated to "indexed"). Then when the application is
> restarted we know that all database records with a status of "indexed"
> exist in the FSDir. This means I can load the FSDir into the InMem dir,
> then update both with any outstanding database records which are
> "pending_index".
>
> Can anyone see any issues with the above system? Is there anything I
> should be aware of when maintaining a RAMDir in parallel to a FSDir?
>
> Regards,
> Joel
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message