hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jagane Sundar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4849) Idempotent create and append operations.
Date Mon, 17 Jun 2013 22:57:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686126#comment-13686126
] 

Jagane Sundar commented on HDFS-4849:
-------------------------------------

Suresh - I hope I am answering the correct comments above. If not, please correct me. Here
goes:

> Client name includes the name of the thread where it was created in. However, one could
use the same FileSystem instance and hence same DFSClient from multiple threads.

As I understand it, by default the FileSystem instance in the client is shared by all threads
in the JVM. Doesn't this imply that the program should either be single threaded, or it should
use its own synchronization of operations between threads? If it is using its own synchronization,
then the above problem will not happen.

If the client really was multi-threaded and desired that each thread should independently
operate on HDFS, it should turn off the FileSystem object caching and use a different FileSystem
object for each thread, should it not?


> Isn't two creates logged in an audit log instead of a single create not a bug?

If the operation is defined as being idempotent, then having two entries in the edits log
is harmless, is it not?
 
                
> Idempotent create and append operations.
> ----------------------------------------
>
>                 Key: HDFS-4849
>                 URL: https://issues.apache.org/jira/browse/HDFS-4849
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>         Attachments: idempotentCreate.patch, idempotentCreate.patch
>
>
> create, append and delete operations can be made idempotent. This will reduce chances
for a job or other app failures when NN fails over.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message