hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rosie Li (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-2257) distcp can copy blocks in parallel
Date Mon, 28 Mar 2011 20:03:07 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Rosie Li updated MAPREDUCE-2257:

    Affects Version/s: 0.21.0
         Release Note: 
copy file parallel by first chopping files into chunks, copy them and stitch the file chunks
back into files in the end.

               Status: Patch Available  (was: Open)

By default, distcp.copy.by.chunk is set to true in the configuration.  The user can set it
to false to use the original distcp. But the type of destination will be checked afterward.
distcp.copy.by.chunk will remain true only if the destination file system is the distributed
file system.

> distcp can copy blocks in parallel
> ----------------------------------
>                 Key: MAPREDUCE-2257
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2257
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: distcp
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
> The minimum unit of work for a distcp task is a file. We have files that are greater
than 1 TB with a block size of  1 GB. If we use distcp to copy these files, the tasks either
take a long long long time or finally fails. A better way for distcp would be to copy all
the source blocks in parallel, and then stich the blocks back to files at the destination
via the HDFS Concat API (HDFS-222)

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message