hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matteo Bertozzi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-7643) HFileArchiver.resolveAndArchive() race condition and snapshot data loss
Date Tue, 22 Jan 2013 00:32:12 GMT
Matteo Bertozzi created HBASE-7643:

             Summary: HFileArchiver.resolveAndArchive() race condition and snapshot data loss
                 Key: HBASE-7643
                 URL: https://issues.apache.org/jira/browse/HBASE-7643
             Project: HBase
          Issue Type: Bug
    Affects Versions: hbase-6055, 0.96.0
            Reporter: Matteo Bertozzi
            Assignee: Matteo Bertozzi
            Priority: Blocker
             Fix For: 0.96.0

 * The master have an hfile cleaner thread (that is responsible for cleaning the /hbase/.archive
 ** /hbase/.archive/table/region/family/hfile
 ** if the family/region/family directory is empty the cleaner removes it
 * The master can archive files (from another thread, e.g. DeleteTableHandler)
 * The region can archive files (from another server/process, e.g. compaction)

The simplified file archiving code looks like this:
HFileArchiver.resolveAndArchive(...) {
  // ensure that the archive dir exists

  // move the file to the archiver
  success = fs.rename(originalPath/fileName, archiveDir/fileName)

  // if the rename is failed, delete the file without archiving
  if (!success) fs.delete(originalPath/fileName);

Since there's no synchronization between HFileArchiver.resolveAndArchive() and the cleaner
run (different process, thread, ...) you can end up in the situation where you are moving
something in a directory that doesn't exists.

// HFileCleaner chore starts at this point
// and the archiveDirectory that we just ensured to be present gets removed.

// The rename at this point will fail since the parent directory is missing.
success = fs.rename(originalPath/fileName, archiveDir/fileName)

The bad thing of deleting the file without archiving is that if you've a snapshot that relies
on the file to be present, or you've a clone table that relies on that file is that you're
losing data.

Possible solutions
 * Create a ZooKeeper lock, to notify the master ("Hey I'm archiving something, wait a bit")
 * Add a RS -> Master call to let the master removes files and avoid this kind of situations
 * Avoid to remove empty directories from the archive if the table exists or is not disabled
 * Add a try catch around the fs.rename

The last one, the easiest one, looks like:
for (int i = 0; i < retries; ++i) {
  // ensure archive directory to be present

  // ----> possible race <-----

  // try to archive file
  success = fs.rename(originalPath/fileName, archiveDir/fileName);
  if (success) break;

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message