hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4248) Renames may cause dangling leases
Date Sat, 01 Dec 2012 00:17:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507783#comment-13507783

Daryn Sharp commented on HDFS-4248:

The code for renaming and leasing handling is problematic.  There's two parallel code paths
for rename, based on whether you call it with src/dest or src/dest/opts.  They are generally
the same copy-n-paste except if the target is a pre-existing directory:  the src/dest method
will implicitly nested the src under the dest, ie. rename /d1 to /d2 produces /d1/d2.  The
src/dest/opts method will fail unless the OVERWRITE option is used - which will replace the
dest, ie. rename /d1 to /d2 will overwrite /d2.

The logic for the replacing or nest into the target is so low level that the callers don't
know the dest was changed.  FSEditLogLoader and FSNameSystem both had to pass a file status
of the dest to change the lease so it could "assume" if the dest was to be nested.  Unfortunately
this causes the src/dest/opt method to have leases incorrectly rewritten, ie. renaming /d1
to /d2 with overwrite produced leases of /d2/d1/d1-contents while the fs had /d2/d1-contents.

As best I can tell, the saving of the namespace is disjointed for finalized inodes, and underconstruction
inodes.  First the finalized inodes are serialized.  Then the path strings in the leases are
used to locate underconstruction inodes for serialization.  This seems a bit fragile, in that
data loss will occur if the lease path goes out of sync with the namespace paths.  I believe
with this design, the state of the leases are so integral to the integrity of the namespace,
that updates to the namespace and leases much occur "atomically" under the same lock at the
same time.

Also, the src/dest/opts method appeared to just "throw away" the src leases resulting in data
loss for open files.

So... What I did:
* changing lease paths only requires src & dest, no more file status of the dest before
the operation
* the namespace and the edit log loader aren't both responsible for the call to update lease
* the leases are updated immediately after the move is successfully
* the src/dest/opts method is not allowed to throw away the source leases

I'm thinking another jira is needed to immediately reclaim leases on src files.  Otherwise
files being written by one user can be moved by another. The updated lease ensures image integrity,
but is useless to the writer because it fails NN operations (path-based) and has no way to
know where the file went and it can't close the file either because the original path no longer
exists and leases are looked up based on path.  The client continues to hold the lease until
it stops actively writing to other streams, which for a daemon might be a really long time.
> Renames may cause dangling leases
> ---------------------------------
>                 Key: HDFS-4248
>                 URL: https://issues.apache.org/jira/browse/HDFS-4248
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Blocker
>         Attachments: HDFS-4248.branch-0.23.patch, HDFS-4248.patch
> Renames of directories may incorrectly re-write the paths in leases under the tree being

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message