hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4849) Idempotent create, append and delete operations.
Date Sat, 25 May 2013 21:42:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13667163#comment-13667163
] 

Todd Lipcon commented on HDFS-4849:
-----------------------------------

The idempotent create is also potentially problematic with multi-threaded clients. A given
client may have multiple threads race to create the same file, and those threads would share
the same client name (and hence lease). Without per-file leases, a second thread trying to
create a file already being created would end up getting back the same block ID and causing
some havoc.

One solution to this would be to include a nonce in the create() call, and store that in the
INodeFileUnderConstruction, so that if you retry with the same nonce, it would identify it
correctly as a retry.

Delete is much harder to make truly idempotent without changing semantics -- I've thought
about it a bit and the only "solution" I could come up with is something like NFS's "duplicate
request cache" - http://www.freesoft.org/CIE/RFC/1813/47.htm -- but we'd have to wire it through
persistently, etc, and it's still somewhat "best effort".
                
> Idempotent create, append and delete operations.
> ------------------------------------------------
>
>                 Key: HDFS-4849
>                 URL: https://issues.apache.org/jira/browse/HDFS-4849
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>
> create, append and delete operations can be made idempotent. This will reduce chances
for a job or other app failures when NN fails over.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message