hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4849) Idempotent create, append and delete operations.
Date Thu, 30 May 2013 07:47:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670136#comment-13670136

Konstantin Shvachko commented on HDFS-4849:


Just to reiterate, create change is not a semantic change, it is an implementation change.
I think it can be done, NN has an internal state to recognize a create retry since it holds
a lease on the file. Should we talk about how it could be implemented?

> How do you guarantee RPC call ID + client name is unique?

Suresh, could you please clarify what you mean by that.
clientName is not guaranteed to be unique right now, unless taskId is unique, which is taken
care of by MR framework, and can be done by other applications.
Call.id (is that the RPC call ID you mentioned) is an incremental counter and will be unique
over the life span of the client.
Are you saying we should make clientName unique?

>  we cache FileSystem objects, many threads can share the same DFSClient.

This is correct. We also have an option to turn off caching, which let's multithreaded apps
use different clients with different names.
Multiple threads using the same cached file system has an issue even now without retries.
Suppose one creates a file and then spawns several threads that start creating blocks simultaneously
on the same file. This will fail.
Todd, is it different from your example?
> Idempotent create, append and delete operations.
> ------------------------------------------------
>                 Key: HDFS-4849
>                 URL: https://issues.apache.org/jira/browse/HDFS-4849
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
> create, append and delete operations can be made idempotent. This will reduce chances
for a job or other app failures when NN fails over.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message