hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jay vyas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5902) Atomic Directory Swapping Operation
Date Fri, 07 Feb 2014 19:52:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13894927#comment-13894927

jay vyas commented on HDFS-5902:

This got me thinking.  The need to unify HDFS logic with other HCFS implementations might
be serviced by a separate JIRA


For example:  Is the atomic-directory-swap logic something which could be managed by a YARN
or ZK based application which any HCFS implementation could plug into? 

> Atomic Directory Swapping Operation
> -----------------------------------
>                 Key: HDFS-5902
>                 URL: https://issues.apache.org/jira/browse/HDFS-5902
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: namenode
>    Affects Versions: 2.2.0
>            Reporter: Benoit Perroud
>            Priority: Minor
> With snapshots, we can already start to have a consistent view of the data over time.
> One missing primitive to do one more step in the direction of a transactional FS is the
ability to swap atomically two folders:
> {code}
> swapFolders(src, dst)
> {code}
> As of today, we need to:
> 1. move the destination folder to a temporary name, 
> 2. rename the source folder in destination's name
> 3. rename the dst folder with the temporary name in the source name.
> This can leads to race conditions where the data is not found.
> An operation inside the namenode achieving folders swapping, with the appropriate checks
and locking to ensure atomicity, would be really useful to perform maintenance tasks like
folder compation, format conversion, etc... (and lot of other use cases).

This message was sent by Atlassian JIRA

View raw message