hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5902) Atomic Directory Swapping Operation
Date Fri, 07 Feb 2014 18:20:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13894803#comment-13894803
] 

Steve Loughran commented on HDFS-5902:
--------------------------------------

I can see the appeal of this, but I can also see that as it wont be available on most filesystems
other than Hadoop. We have problems with code that expects that rm -r and mv are atomic, even
mkdirs().

if you were to do it, 

# it'd have to be an interface that FileSystems may support, but which isn't there if it is
not (blobstores would leave it out, localfs, etc)
# it must be within the same parent dir to work with filesystems that mount bits in the tree
(including federation?) 
{code}
   swapFolders(parent, srcchild, destchild
{code}

> Atomic Directory Swapping Operation
> -----------------------------------
>
>                 Key: HDFS-5902
>                 URL: https://issues.apache.org/jira/browse/HDFS-5902
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: namenode
>    Affects Versions: 2.2.0
>            Reporter: Benoit Perroud
>            Priority: Minor
>
> With snapshots, we can already start to have a consistent view of the data over time.
> One missing primitive to do one more step in the direction of a transactional FS is the
ability to swap atomically two folders:
> {code}
> swapFolders(src, dst)
> {code}
> As of today, we need to:
> 1. move the destination folder to a temporary name, 
> 2. rename the source folder in destination's name
> 3. rename the dst folder with the temporary name in the source name.
> This can leads to race conditions where the data is not found.
> An operation inside the namenode achieving folders swapping, with the appropriate checks
and locking to ensure atomicity, would be really useful to perform maintenance tasks like
folder compation, format conversion, etc... (and lot of other use cases).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message