hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4974) Analyze and add annotations to Namenode and Datanode protocol methods to enable retry
Date Mon, 15 Jul 2013 20:46:50 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13708957#comment-13708957

Suresh Srinivas commented on HDFS-4974:

bq. Holding the write lock keeps the operation thread-safe (multiple concurrent calls won't
corrupt state), but not idempotent, because repeated calls yield side effects (one copy of
the data in the file vs. multiple).
What I mean by idempotent is - currently metasave appends some information to a log file.
If the lock was not held, it is possible to have interleaved/incorrect information in metasave.
That is not the case here. All you end up having is an additional copy of metasave in the
file. That should not result in issues is what I am thinking.

bq. I suggest that we switch to opening the output file with overwrite instead of append to
make it truly idempotent. If that change is seen as too risky, then we could apply the retry
cache to this call.
I think this seems reasonable. But it is a changed behavior we need to document as well. Would
you like to take up that up?
> Analyze and add annotations to Namenode and Datanode protocol methods to enable retry
> -------------------------------------------------------------------------------------
>                 Key: HDFS-4974
>                 URL: https://issues.apache.org/jira/browse/HDFS-4974
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha, namenode
>            Reporter: Suresh Srinivas
> This jira is intended for:
> # Discussing current @Idempotent annotations in HDFS protocols and adding that annotation
where it is missing.
> # Discuss how retry should be enabled for non-idempotent requests.
> I will post the analysis of current methods in subsequent comment.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message