hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Omkar Aradhya K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11794) distcp can copy blocks in parallel
Date Tue, 21 Mar 2017 03:44:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934045#comment-15934045
] 

Omkar Aradhya K S commented on HADOOP-11794:
--------------------------------------------

I was trying to evaluate your patch with ADLS:
Tried the bits on a HDInsight 3.5 cluster (this comes with hadoop 2.7)
Observed following compatibility issues:
     a.	You are checking for instance of {code}DistributedFileSystem{code} in many places
and all other {code}FileSystem{code} implementations don’t implement {code}DistributedFileSystem{code}
          i.	Could this be changed to something more compatible with other {code}FileSystem{code}
implementations?
     b.	You are using the new {code}DFSUtilClient{code}, which makes DistCp incompatible with
older versions of Hadoop
         i.	Can this be changed to be backward compatible?
If the compatibility issues are addressed, the DistCp with your feature would be available
for other {code}FileSystem{code} implementations and also would be backward compatible.

> distcp can copy blocks in parallel
> ----------------------------------
>
>                 Key: HADOOP-11794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: tools/distcp
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Yongjun Zhang
>         Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch, HADOOP-11794.003.patch,
HADOOP-11794.004.patch, HADOOP-11794.005.patch, HADOOP-11794.006.patch, HADOOP-11794.007.patch,
HADOOP-11794.008.patch, MAPREDUCE-2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that are greater
than 1 TB with a block size of  1 GB. If we use distcp to copy these files, the tasks either
take a long long long time or finally fails. A better way for distcp would be to copy all
the source blocks in parallel, and then stich the blocks back to files at the destination
via the HDFS Concat API (HDFS-222)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message