hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4849) Idempotent create, append and delete operations.
Date Wed, 29 May 2013 04:56:23 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668993#comment-13668993
] 

Konstantin Shvachko commented on HDFS-4849:
-------------------------------------------

Given Matthew's comment I think I should have provided more motivation for the issue first.
The idea to make these changes comes from the desire to have MR and YARN run the jobs without
interruption in HA case.
Today if NameNode dies and failover to StandbyNode occurs some jobs can fail. This mostly
depends on whether failure of NN happened during idempotent or non-idempotent operation. Idempotent
operations, like getBlockLocations or addBlock, are retried and the client will eventually
complete such operation via StandbyNode, when SBN becomes active. Non-idempotent operations
like create and delete are not retried, they just fail. Therefore, MR job fails if it tries
to create an output file for a reducer or delete a directory at cleanup stage just at the
moment NN crashes. While if it could retry the create on SBN, it would have succeeded.
So we might need to compromize and loozen the semantics of some HDFS operations in order to
satisfy stricter availabilty and scalability requirements. And we better do it now before
APIs are frozen for branch 2.
                
> Idempotent create, append and delete operations.
> ------------------------------------------------
>
>                 Key: HDFS-4849
>                 URL: https://issues.apache.org/jira/browse/HDFS-4849
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>
> create, append and delete operations can be made idempotent. This will reduce chances
for a job or other app failures when NN fails over.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message