lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <mar...@rectangular.com>
Subject Re: [lucy-user] C library, how to check index is healthy
Date Tue, 14 Feb 2017 18:36:24 GMT
On Tue, Feb 14, 2017 at 7:43 AM, Tilghman Lesher <tilghman@meg.abyt.es> wrote:

> As another sidenote, there are techniques for reliable exclusive
> locking when the datastore is NFS.  Namely, instead of using the
> default locking mechanisms in Unix, you can use the link(2) system
> interface (which is an atomic operation on NFS) with an agreed-upon
> name for your lock.  For example, if your shared volume was "/shared",
> then you could create a temporary file using mkstemp on the volume,
> then attempt to link(2) the temporary file to that known lockfile
> name, "/shared/lock".  If the link succeeds, you have the lock, but if
> the operation fails, another process obtained the lock.

That is, in fact, what Lucy does internally.

    https://github.com/apache/lucy/blob/rel/v0.6.1/core/Lucy/Store/Lock.c#L188

    // Write to a temporary file, then use the creation of a hard link to
    // ensure atomic but non-destructive creation of the lockfile with its
    // complete contents.

> This method
> does require that your processes clean up (i.e. delete) the file when
> you want to release the lock, however.

Right, and we have some logic to clean up the lockfile automatically.  The
lock file contains a host name and a PID; if the host name matches AND the pid
is not active, we assume that the lockfile can be deleted.

This default behavior works pretty well for "typical" use on normal local
volumes -- it deletes many stale lockfiles automatically and generally spares
users from having to evaluate whether they need to do it themselves.  The
price is that on NFS and the like you typically need to override the default:
to be safe when you have multiple machines trying to write to an index on a
shared volume, you must ensure that each Indexer is associated with the proper
host name (via IndexManager).

(For more info, see http://lucy.apache.org/docs/c/Lucy/Docs/FileLocking .)

> 1. Build to /shared/index_123/ (number could also be the PID of the
> index-building process).
> 2. Delete /shared/index_old/.
> 3. Use readlink(2) to grab the current (real) pathname of the index
> (/shared/index_122)
> 4. cd /shared ; ln -sf index_123/ /shared/index (production path)
> 5. Rename the previous index (/shared/index_122) to /shared/index_old/.
>
> By building the index under a temporary directory name, then swapping
> out the directory when we want to put the new index into production,
> we avoid the locking problems between readers and writers entirely.

I can see how this works, though it is costly if you're building the indexes
from scratch each time rather than taking advantage of Lucy's incremental
indexing.

To speed things up, Lucy could supply a way to copy an entire index
near-instantaneously using hard links.  (This works because index files, once
committed, are never modified -- index content only changes through the
addition of new files and eventual deletion of obsolete files.)  The interface
could look something like this:

    lucy_Backup *backup = lucy_Backup_new("/path/to/index");
    cfish_String *snapshot_name = Lucy_Backup_Get_Snapshot_Name);
    cfish_String *backup_path
        = cfish_String_newf("/backupdir/backup_%o", snapshot_name);
    Lucy_Backup_Hard_Link_Dupe(backup, backup_path);

Then the following workflow becomes possible:

1. Use `hard_link_dupe` to create a duplicate index.
2. Add new content to the duped index
3. ln -sf /shared/index_123 /shared/index (production path)
4. Remove old indexes after some timeout.  (All searchers must be refreshed
   on a schedule which guarantees they do not access deleted content, or
   you'll see `Stale NFS filehandle` exceptions.)

Marvin Humphrey

Mime
View raw message