hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4974) Analyze and add annotations to Namenode and Datanode protocol methods to enable retry
Date Mon, 15 Jul 2013 19:54:49 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13708884#comment-13708884

Chris Nauroth commented on HDFS-4974:

Given the entire metasave is done with writeLock() held, will this be an issue? Currently
one could have two metasaves issued by two clients right? Given that I think it should be
okay to mark it as idempotent?

Holding the write lock keeps the operation thread-safe (multiple concurrent calls won't corrupt
state), but not idempotent, because repeated calls yield side effects (one copy of the data
in the file vs. multiple).

It is correct that 2 clients requesting a metasave to the same output file would also cause
this problem.  This is slightly different in that it's not retries of the exact same RPC call.

I suggest that we switch to opening the output file with overwrite instead of append to make
it truly idempotent.  If that change is seen as too risky, then we could apply the retry cache
to this call.
> Analyze and add annotations to Namenode and Datanode protocol methods to enable retry
> -------------------------------------------------------------------------------------
>                 Key: HDFS-4974
>                 URL: https://issues.apache.org/jira/browse/HDFS-4974
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha, namenode
>            Reporter: Suresh Srinivas
> This jira is intended for:
> # Discussing current @Idempotent annotations in HDFS protocols and adding that annotation
where it is missing.
> # Discuss how retry should be enabled for non-idempotent requests.
> I will post the analysis of current methods in subsequent comment.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message