hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rosie Li (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-2257) distcp can copy blocks in parallel
Date Thu, 31 Mar 2011 17:56:05 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rosie Li updated MAPREDUCE-2257:
--------------------------------

    Release Note: 
copy file parallel by first chopping files into chunks, copy them and stitch the file chunks
back into files in the end.
By default, distcp.copy.by.chunk is set to true in the configuration. The user can set it
to false to use the original distcp. But the type of destination will be checked afterward.
distcp.copy.by.chunk will remain true only if the destination file system is the distributed
file system(where we can use the concat method to stitch the file chunks together in the end)


  was:
copy file parallel by first chopping files into chunks, copy them and stitch the file chunks
back into files in the end.




> distcp can copy blocks in parallel
> ----------------------------------
>
>                 Key: MAPREDUCE-2257
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2257
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: distcp
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: MAPREDUCE-2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that are greater
than 1 TB with a block size of  1 GB. If we use distcp to copy these files, the tasks either
take a long long long time or finally fails. A better way for distcp would be to copy all
the source blocks in parallel, and then stich the blocks back to files at the destination
via the HDFS Concat API (HDFS-222)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message