hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4258) Rename of Being Written Files
Date Thu, 21 Feb 2013 22:52:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583638#comment-13583638

Aaron T. Myers commented on HDFS-4258:

bq. I have hard time understanding the relationship of that with rename in this jira. In fact
when recoverLease is done an active writer will not be able to write whether rename has occurred
or not. That behavior is not being changed with the changes done as a part of this.

I wasn't implying that this proposed change has anything to do with the recoverLease API.
I was just using it as a counterpoint to Brandon's claim that clients would be overpowered
since "for any client wants to gain a lease of any file, it could just rename it and then
open it. Most likely it could get the lease immediately as long as it has write permission."
i.e. that should not be a concern, since a client can already do that today with the recoverLease

There are other ideas that have been proposed about revoking the lease on rename. I am -1
on it for the following reasons:
# Current behavior is when a rename occurs the current writer continues to write to the block
that is currently allocated but fails to allocate new blocks.
# New rename behavior will be incompatible where the current writer is fenced from writing.
Given that a lot of files in HDFS are less than a block in length, this could result in strange
behaviors for some applications.
# I agree with the point Brandon raised above. Renaming a directory means walking through
all the files open under it and revoking the leases. Rename already is a complicated operation.
Doing this additional work during rename makes it even more heavier and the operation unpredictably

I actually like the direction Brandon and Nicholas have taken. We can continue the existing
behavior. In fact with this change, we can allow current writer to continue to allocated new
blocks (based on file ID) and continue to write, if we want. But that could be done in another

I agree with Daryn's reasoning as stated in HDFS-4437:

I think supporting file descriptor behavior is a great idea (we've internally talked about
this). Until we do, I think the lease should be revoked. My concerns with fd behavior would
be the ever pervasive "two wrongs make a right" where users are relying unintentionally on
renames breaking writers, and ensuring we get the security right to avoid attacks probing
for fileids.

Regardless of whether or not we implement HDFS-4437 as Daryn proposed, I still think we should
move the INode ID stuff to a branch. It's a fairly involved change with several sub-task JIRAs,
which indicates to me that it would be better done incrementally on a branch and then merged
to trunk once it's a completely functional whole.
> Rename of Being Written Files
> -----------------------------
>                 Key: HDFS-4258
>                 URL: https://issues.apache.org/jira/browse/HDFS-4258
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client, namenode
>    Affects Versions: 3.0.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Brandon Li
>         Attachments: HDFS-4258.patch, HDFS-4258.patch, HDFS-4258.patch, HDFS-4258.patch
> When a being written file or it's ancestor directories is renamed, the path in the file
lease is also renamed.  Then the writer of the file usually will fail since the file path
in the writer is not updated.
> Moreover, I think there is a bug as follow:
> # Client writes 0's to F_0="/foo/file" and writes 1's to F_1="/bar/file" at the same
> # Rename /bar to /baz
> # Rename /foo to /bar
> Then, writing to F_0 will fail since /foo/file does not exist anymore but writing to
F_1 may succeed since /bar/file exits as a different file.  In such case, the content of /bar/file
could be partly 0's and partly 1's.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message