Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 21 Mar 2017 03:44:41 +0000 (UTC)
From: "Omkar Aradhya K S (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12495281.1294770616000.74430.1490067881778@Atlassian.JIRA>
In-Reply-To: <JIRA.12495281.1294770616000@Atlassian.JIRA>
References: <JIRA.12495281.1294770616000@Atlassian.JIRA> <JIRA.12495281.1294770616277@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-11794) distcp can copy blocks in
 parallel
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Tue, 21 Mar 2017 03:44:47 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-11794?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15=
934045#comment-15934045 ]=20

Omkar Aradhya K S commented on HADOOP-11794:
--------------------------------------------

I was trying to evaluate your patch with ADLS:
Tried the bits on a HDInsight 3.5 cluster (this comes with hadoop 2.7)
Observed following compatibility issues:
     a.=09You are checking for instance of {code}DistributedFileSystem{code=
} in many places and all other {code}FileSystem{code} implementations don=
=E2=80=99t implement {code}DistributedFileSystem{code}
          i.=09Could this be changed to something more compatible with othe=
r {code}FileSystem{code} implementations?
     b.=09You are using the new {code}DFSUtilClient{code}, which makes Dist=
Cp incompatible with older versions of Hadoop
         i.=09Can this be changed to be backward compatible?
If the compatibility issues are addressed, the DistCp with your feature wou=
ld be available for other {code}FileSystem{code} implementations and also w=
ould be backward compatible.

> distcp can copy blocks in parallel
> ----------------------------------
>
>                 Key: HADOOP-11794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: tools/distcp
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Yongjun Zhang
>         Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch, HADO=
OP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch, HADOOP-=
11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch, MAPREDUCE-=
2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that =
are greater than 1 TB with a block size of  1 GB. If we use distcp to copy =
these files, the tasks either take a long long long time or finally fails. =
A better way for distcp would be to copy all the source blocks in parallel,=
 and then stich the blocks back to files at the destination via the HDFS Co=
ncat API (HDFS-222)


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org