incubator-lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <>
Subject Snapshot: "segments" key
Date Fri, 26 Mar 2010 21:04:46 GMT

Snapshot is currently "a list of files" and nothing more.  Its interface is
powerful because it allows subcomponents to manage their own files.  However,
it's also a bit of a pain, because subcomponents *must* manage their own files.  

When a segment gets deleted, Snapshot doesn't understand that the whole
directory and all of its contents is going away.  It must be told about each
file within the directory, which means that every subcomponent must implement
Delete_Segment() instead of relying on a no-op default.

To simplify the subcomponent code, I think we should make a change to
Snapshot, and add a new key, "segments".  Here's the contents of a current
snapshot.json file:

       "entries" : [ 
       "format" : "1" 

After the change, it would look like this:

       "entries" : [ 
       "segments" : [
       "format" : "1" 

When a segment goes away, in addition to having FilePurger call Folder_Delete()
for each element within the "entries" array, we can have it call
Folder_Delete_Tree() on each element within "segments".  The recursive deletion
will relieve us of the need to track files within those directories.

I considered a couple other possibilities before arriving at this plan.  

We could just call Delete_Tree() on every element within "entries" -- but that
seemed too heavy handed and inflexible.

We could have a "directories" key instead of a "segments" key, but that seemed
strange.  It would make more sense to have "directories" and "files" instead of
"directories" and "entries" -- however, I'm not sure that it makes sense to
have a rule that we always recursively delete "directories".  In contrast,
segment directories are special within Lucy, so I think it makes sense to
special-case those.

I expect to make this change to KinoSearch shortly after we release 0.30_09,
as the first in a a flurry of small file format changes for 0.30_10.
Subsequently, I'll open up a JIRA ticket and add the missing Write_File() and
Read_File() methods to Lucy's Snapshot.

Marvin Humphrey

View raw message