hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4849) Idempotent create and append operations.
Date Tue, 18 Jun 2013 17:56:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687005#comment-13687005

Konstantin Shvachko commented on HDFS-4849:

> Please wait till the ongoing discussion ends.

I am not in a hurry. I can wait as long as this change is in 2.1.

> Without your change, all the subsequent fs.create() will fail right?

No. Could you please try to run it. 
It has a main, you just need to past&copy it into your favourite HDFS test and run as
"Java application" if you still use Eclipse.

In the mean time let's talk about idempotency.
What is your definition of an idempotent operation? How is it different from "just allowing

Here is the definition I know. Create is idempotent iff {color:orange}_create(path, NS) =
create(path, create(path, NS))_{color}
If operation is idempotent it allows retries, which is the main reason we want operations
to be idempotent.
This patch does not build a retry cache, it uses existing information about the file to recognize
a retry. 
This follows the same logic as addBlock(), which recognizes a retry based on the file state
and its input parameters.
So I think my patch provides idempotent create and append.
> Idempotent create and append operations.
> ----------------------------------------
>                 Key: HDFS-4849
>                 URL: https://issues.apache.org/jira/browse/HDFS-4849
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>         Attachments: idempotentCreate.patch, idempotentCreate.patch, idempotentCreate.patch
> create, append and delete operations can be made idempotent. This will reduce chances
for a job or other app failures when NN fails over.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message