hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-4872) Idempotent delete operation.
Date Tue, 04 Jun 2013 21:09:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13673589#comment-13673589
] 

Konstantin Shvachko edited comment on HDFS-4872 at 6/4/13 9:08 PM:
-------------------------------------------------------------------

This discussion started in HDFS-4849.
Where we discussed -four- five approaches so far

# Just mark delete idempotent. 
[A delete retry may delete an object that has been recreated or replaced between the retries
in this case.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670142&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670142]
# Design ["duplicate request cache"|http://www.freesoft.org/CIE/RFC/1813/47.htm].
This is applicable to almost any operation and semantics. Drawbacks: complexity, performance,
and RAM overhead.
[See last two paragraphs.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670538&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670538]
# Add modTime parameter to delete operation with the meaning that the object is deleted only
if its modification time is <= than modTime parameter.
# Replace delete with idempotent rename to a temporary object, then delete the latter with
non-idempotent delete.
[See the beginning of this comment.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670538&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670538]
# [Delete by inodeID.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13669018&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13669018]
This is a two step delete: first obtain inodeID, then call delete(path, inodeID) with a meaning
that the path is deleted if the object referred still has the same inodeID. 
                
      was (Author: shv):
    This discussion started in HDFS-4849.
Where we discussed four approaches so far

# Just mark delete idempotent. 
[A delete retry may delete an object that has been recreated or replaced between the retries
in this case.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670142&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670142]
# Design ["duplicate request cache"|http://www.freesoft.org/CIE/RFC/1813/47.htm].
This is applicable to almost any operation and semantics. Drawbacks: complexity, performance,
and RAM overhead.
[See last two paragraphs.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670538&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670538]
# Add modTime parameter to delete operation with the meaning that the object is deleted only
if its modification time is <= than modTime parameter.
# Replace delete with idempotent rename to a temporary object, then delete the latter with
non-idempotent delete.
[See the beginning of this comment.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670538&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670538]
                  
> Idempotent delete operation.
> ----------------------------
>
>                 Key: HDFS-4872
>                 URL: https://issues.apache.org/jira/browse/HDFS-4872
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>
> Making delete idempotent is important to provide uninterrupted job execution in case
of HA failover.
> This is to discuss different approaches to idempotent implementation of delete.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message