hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4849) Idempotent create and append operations.
Date Tue, 18 Jun 2013 16:46:24 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686910#comment-13686910

Suresh Srinivas commented on HDFS-4849:

bq. I don't see any technical obstacles to committing this based on Suresh's comments. Suresh,
is it OK to commit?
I am surprised that, despite my comments in https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13686381&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13686381
you are saying you do not see "technical obstacles"! Is this because of cross posting. Please
wait till the ongoing discussion ends. I also have few code review comments, I will post that
after the discussion is over.

bq. With idempotent creation operation such a reconstructions would be perfectly sane:
I have explained earlier why this change does not make create and append idempotent. Other
methods tagged by [~atm] were indeed idempotent. This only changes the methods so that retry
from the same client is allowed. Lets first agree that this is change is not making create
and append idempotent.

bq. As for the change in behavior: isn't what the beta is about? To land the changes before
the stabilization?
The reason why I am asking this question is - current patch is sort of building retry cache
for create and append using lease information. That means, on retry, namespace is not change,
editlog does not additional log. Only the method returns what would have been the result on
the first try. Given it works like retry cache, adding an entry into audit log seems unnecessary.
In case of other operations marked idempotent, it indeed performs all the necessary changes
(including editlog entry) on the namespace and hence auditlog there makes sense.

bq. The snippet below shows that multiple creates are allowed in current code, and the things
break because there is a competition for block allocations not file creates.
I do not understand this comment. Without your change, all the subsequent fs.create() will
fail right? So other than one thread that succeeds in create(), all the other threads should
get FileAlreadyExistsException right?

> Idempotent create and append operations.
> ----------------------------------------
>                 Key: HDFS-4849
>                 URL: https://issues.apache.org/jira/browse/HDFS-4849
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>         Attachments: idempotentCreate.patch, idempotentCreate.patch, idempotentCreate.patch
> create, append and delete operations can be made idempotent. This will reduce chances
for a job or other app failures when NN fails over.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message