hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7643) HFileArchiver.resolveAndArchive() race condition may lead to snapshot data loss
Date Thu, 24 Jan 2013 08:39:13 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13561500#comment-13561500

Hadoop QA commented on HBASE-7643:

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 3 new or modified

    {color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 2.0 profile.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:red}-1 findbugs{color}.  The patch appears to introduce 1 new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 lineLengths{color}.  The patch does not introduce lines longer than 100

    {color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4156//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4156//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4156//console

This message is automatically generated.
> HFileArchiver.resolveAndArchive() race condition may lead to snapshot data loss
> -------------------------------------------------------------------------------
>                 Key: HBASE-7643
>                 URL: https://issues.apache.org/jira/browse/HBASE-7643
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: hbase-6055, 0.96.0
>            Reporter: Matteo Bertozzi
>            Assignee: Matteo Bertozzi
>            Priority: Blocker
>             Fix For: 0.96.0, 0.94.5
>         Attachments: HBASE-7653-p4-v0.patch, HBASE-7653-p4-v1.patch, HBASE-7653-p4-v2.patch,
HBASE-7653-p4-v3.patch, HBASE-7653-p4-v4.patch, HBASE-7653-p4-v5.patch, HBASE-7653-p4-v6.patch
>  * The master have an hfile cleaner thread (that is responsible for cleaning the /hbase/.archive
>  ** /hbase/.archive/table/region/family/hfile
>  ** if the family/region/family directory is empty the cleaner removes it
>  * The master can archive files (from another thread, e.g. DeleteTableHandler)
>  * The region can archive files (from another server/process, e.g. compaction)
> The simplified file archiving code looks like this:
> {code}
> HFileArchiver.resolveAndArchive(...) {
>   // ensure that the archive dir exists
>   fs.mkdir(archiveDir);
>   // move the file to the archiver
>   success = fs.rename(originalPath/fileName, archiveDir/fileName)
>   // if the rename is failed, delete the file without archiving
>   if (!success) fs.delete(originalPath/fileName);
> }
> {code}
> Since there's no synchronization between HFileArchiver.resolveAndArchive() and the cleaner
run (different process, thread, ...) you can end up in the situation where you are moving
something in a directory that doesn't exists.
> {code}
> fs.mkdir(archiveDir);
> // HFileCleaner chore starts at this point
> // and the archiveDirectory that we just ensured to be present gets removed.
> // The rename at this point will fail since the parent directory is missing.
> success = fs.rename(originalPath/fileName, archiveDir/fileName)
> {code}
> The bad thing of deleting the file without archiving is that if you've a snapshot that
relies on the file to be present, or you've a clone table that relies on that file is that
you're losing data.
> Possible solutions
>  * Create a ZooKeeper lock, to notify the master ("Hey I'm archiving something, wait
a bit")
>  * Add a RS -> Master call to let the master removes files and avoid this kind of
>  * Avoid to remove empty directories from the archive if the table exists or is not disabled
>  * Add a try catch around the fs.rename
> The last one, the easiest one, looks like:
> {code}
> for (int i = 0; i < retries; ++i) {
>   // ensure archive directory to be present
>   fs.mkdir(archiveDir);
>   // ----> possible race <-----
>   // try to archive file
>   success = fs.rename(originalPath/fileName, archiveDir/fileName);
>   if (success) break;
> }
> {code}

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message