hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4849) Idempotent create, append and delete operations.
Date Sat, 25 May 2013 21:42:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13667163#comment-13667163

Todd Lipcon commented on HDFS-4849:

The idempotent create is also potentially problematic with multi-threaded clients. A given
client may have multiple threads race to create the same file, and those threads would share
the same client name (and hence lease). Without per-file leases, a second thread trying to
create a file already being created would end up getting back the same block ID and causing
some havoc.

One solution to this would be to include a nonce in the create() call, and store that in the
INodeFileUnderConstruction, so that if you retry with the same nonce, it would identify it
correctly as a retry.

Delete is much harder to make truly idempotent without changing semantics -- I've thought
about it a bit and the only "solution" I could come up with is something like NFS's "duplicate
request cache" - http://www.freesoft.org/CIE/RFC/1813/47.htm -- but we'd have to wire it through
persistently, etc, and it's still somewhat "best effort".
> Idempotent create, append and delete operations.
> ------------------------------------------------
>                 Key: HDFS-4849
>                 URL: https://issues.apache.org/jira/browse/HDFS-4849
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
> create, append and delete operations can be made idempotent. This will reduce chances
for a job or other app failures when NN fails over.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message