Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Sat, 1 Dec 2012 00:17:58 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <1985050120.47884.1354321078321.JavaMail.jiratomcat@arcas>
In-Reply-To: <1522097891.45429.1354290478654.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (HDFS-4248) Renames may cause dangling leases
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507783#comment-13507783 ] 

Daryn Sharp commented on HDFS-4248:
-----------------------------------

The code for renaming and leasing handling is problematic.  There's two parallel code paths for rename, based on whether you call it with src/dest or src/dest/opts.  They are generally the same copy-n-paste except if the target is a pre-existing directory:  the src/dest method will implicitly nested the src under the dest, ie. rename /d1 to /d2 produces /d1/d2.  The src/dest/opts method will fail unless the OVERWRITE option is used - which will replace the dest, ie. rename /d1 to /d2 will overwrite /d2.

The logic for the replacing or nest into the target is so low level that the callers don't know the dest was changed.  FSEditLogLoader and FSNameSystem both had to pass a file status of the dest to change the lease so it could "assume" if the dest was to be nested.  Unfortunately this causes the src/dest/opt method to have leases incorrectly rewritten, ie. renaming /d1 to /d2 with overwrite produced leases of /d2/d1/d1-contents while the fs had /d2/d1-contents.

As best I can tell, the saving of the namespace is disjointed for finalized inodes, and underconstruction inodes.  First the finalized inodes are serialized.  Then the path strings in the leases are used to locate underconstruction inodes for serialization.  This seems a bit fragile, in that data loss will occur if the lease path goes out of sync with the namespace paths.  I believe with this design, the state of the leases are so integral to the integrity of the namespace, that updates to the namespace and leases much occur "atomically" under the same lock at the same time.

Also, the src/dest/opts method appeared to just "throw away" the src leases resulting in data loss for open files.

So... What I did:
* changing lease paths only requires src & dest, no more file status of the dest before the operation
* the namespace and the edit log loader aren't both responsible for the call to update lease paths
* the leases are updated immediately after the move is successfully
* the src/dest/opts method is not allowed to throw away the source leases

I'm thinking another jira is needed to immediately reclaim leases on src files.  Otherwise files being written by one user can be moved by another. The updated lease ensures image integrity, but is useless to the writer because it fails NN operations (path-based) and has no way to know where the file went and it can't close the file either because the original path no longer exists and leases are looked up based on path.  The client continues to hold the lease until it stops actively writing to other streams, which for a daemon might be a really long time.
                
> Renames may cause dangling leases
> ---------------------------------
>
>                 Key: HDFS-4248
>                 URL: https://issues.apache.org/jira/browse/HDFS-4248
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Blocker
>         Attachments: HDFS-4248.branch-0.23.patch, HDFS-4248.patch
>
>
> Renames of directories may incorrectly re-write the paths in leases under the tree being moved.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira