hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen He (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-12046) Avoid creating "._COPYING_" temporary file when copying file to Swift file system
Date Mon, 22 Jun 2015 05:42:01 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chen He updated HADOOP-12046:
-----------------------------
    Attachment: 屏幕快照 2015-06-21 下午10.38.34.png

Attach the file copy process if user tries to copy a file(larger than 5GB) from HDFS to Swift
using current Swift driver. 

> Avoid creating "._COPYING_" temporary file when copying file to Swift file system
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-12046
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12046
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/swift
>    Affects Versions: 2.7.0
>            Reporter: Chen He
>            Assignee: Chen He
>
> When copy file from HDFS or local to another file system implementation, in CommandWithDestination.java,
it creates a temp file by adding suffix "._COPYING_". Once file is successfully copied, it
will remove the suffix by rename(). 
> try {
>       PathData tempTarget = target.suffix("._COPYING_");
>       targetFs.setWriteChecksum(writeChecksum);
>       targetFs.writeStreamToFile(in, tempTarget, lazyPersist);
>       targetFs.rename(tempTarget, target);
>     } finally {
>       targetFs.close(); // last ditch effort to ensure temp file is removed
>     }
> It is not costly in HDFS. However, if copy to Swift file system, the rename process is
to create a new file. It is not efficient if users copy a lot of files to swift file system.
I did some tests, for a 1G file copying to swift, it will take 10% more time. We should only
do the copy one time for Swift file system. Changes should be limited to the Swift driver
level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message