Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 294131101D for ; Sat, 20 Sep 2014 20:12:12 +0000 (UTC) Received: (qmail 51258 invoked by uid 500); 20 Sep 2014 20:12:04 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 51135 invoked by uid 500); 20 Sep 2014 20:12:04 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 51125 invoked by uid 99); 20 Sep 2014 20:12:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Sep 2014 20:12:04 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jameel@6sense.com designates 157.56.111.57 as permitted sender) Received: from [157.56.111.57] (HELO na01-bn1-obe.outbound.protection.outlook.com) (157.56.111.57) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Sep 2014 20:11:59 +0000 Received: from CO2PR0601MB0965.namprd06.prod.outlook.com (25.160.10.25) by CO2PR0601MB0965.namprd06.prod.outlook.com (25.160.10.25) with Microsoft SMTP Server (TLS) id 15.0.1034.13; Sat, 20 Sep 2014 20:11:35 +0000 Received: from CO2PR0601MB0965.namprd06.prod.outlook.com ([25.160.10.25]) by CO2PR0601MB0965.namprd06.prod.outlook.com ([25.160.10.25]) with mapi id 15.00.1034.003; Sat, 20 Sep 2014 20:11:35 +0000 From: Jameel Al-Aziz To: "user@hadoop.apache.org" Subject: Re: Unable to use transfer data using distcp between EC2-classic cluster and VPC cluster Thread-Topic: Unable to use transfer data using distcp between EC2-classic cluster and VPC cluster Thread-Index: AQHP1Gw9OPCY94NB2kCr8iK51VZ0gJwKH/iAgAAFIICAAFA2oQ== Date: Sat, 20 Sep 2014 20:11:34 +0000 Message-ID: <481e80d5-e45f-464a-b991-a9d5ec955cf9@6sense.com> References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [76.217.208.122] x-microsoft-antispam: BCL:0;PCL:0;RULEID:;SRVR:CO2PR0601MB0965; x-forefront-prvs: 0340850FCD x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(189002)(57704003)(53754006)(199003)(41574002)(24454002)(377454003)(31686004)(33646002)(2501002)(16236675004)(19617315012)(15395725005)(106116001)(101416001)(92566001)(2351001)(107886001)(107046002)(15202345003)(85306004)(110136001)(97736003)(105586002)(83322001)(76482002)(19580405001)(19580395003)(76176999)(64706001)(50986999)(54356999)(66066001)(31696002)(90102001)(31966008)(99396002)(21056001)(106356001)(36756003)(86362001)(85852003)(20776003)(77096002)(83072002)(81342003)(81542003)(16601075003)(74662003)(2656002)(79102003)(46102003)(77982003)(74502003)(80022003)(87936001)(92726001)(99286002)(4396001)(217423001)(15975445006)(95666004)(21314002);DIR:OUT;SFP:1101;SCL:1;SRVR:CO2PR0601MB0965;H:CO2PR0601MB0965.namprd06.prod.outlook.com;FPR:;MLV:sfv;PTR:InfoNoRecords;MX:1;A:1;LANG:en; Content-Type: multipart/alternative; boundary="_000_481e80d5e45f464ab991a9d5ec955cf96sensecom_" MIME-Version: 1.0 X-OriginatorOrg: 6sense.com X-Virus-Checked: Checked by ClamAV on apache.org --_000_481e80d5e45f464ab991a9d5ec955cf96sensecom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Ankit, We originally tried to copy to S3 and back. In fact, it is actually our fal= lback plan. We were having issues with the copy to S3 not maintaining the d= irectory layout, so we decided to try and do a direct copy. I'll give it another shot though! Jameel Al-Aziz From: Ankit Singhal Sent: Sep 20, 2014 8:25 AM To: user@hadoop.apache.org Subject: Re: Unable to use transfer data using distcp between EC2-classic c= luster and VPC cluster Hi Jameel, As Peyman said, best approach is to do distcp from your old cluster to s3 a= nd making MR job reading directly from s3 on new cluster. but If you still need to do distcp from hdfs to hdfs then update /etc/hosts= or DNS of all the nodes of your old cluster with "publicIp internalAWSDN= SName" of all nodes of new cluster. for eq:- /etc/hosts of all nodes of old cluster should have entry of all the nodes o= f new cluster in below format. 54.xxx.xxx.xx1 ip-10-xxx-xxx-xx1.ec2.internal 54.xxx.xxx.xx2 ip-10-xxx-xxx-xx2.ec2.internal 54.xxx.xxx.xx3 ip-10-xxx-xxx-xx3.ec2.internal Regards, Ankit Singhal On Sat, Sep 20, 2014 at 8:36 PM, Peyman Mohajerian > wrote: It maybe easier to copy the data to s3 and then from s3 to the new cluster. On Fri, Sep 19, 2014 at 8:45 PM, Jameel Al-Aziz > wrote: Hi all, We're in the process of migrating from EC2-Classic to VPC and needed to tra= nsfer our HDFS data. We setup a new cluster inside the VPC, and assigned th= e name node and data node temporary public IPs. Initially, we had a lot of = trouble getting the name node to redirect to the public hostname instead of= private IPs. After some fiddling around, we finally got webhdfs and dfs -c= p to work using public hostnames. However, distcp simply refuses to use the= public hostnames when connecting to the data nodes. We're running distcp on the old cluster, copying data into the new cluster. The old hadoop cluster is running 1.0.4 and the new one is running 1.2.1. So far, on the new cluster, we've tried: - Using public DNS hostnames in the master and slaves files (on both the na= me node and data nodes) - Setting the hostname of all the boxes to their public DNS name - Setting "fs.default.name" to the public DNS name = of the new name node. And on both clusters: - Setting the "dfs.datanode.use.datanode.hostname" and "dfs.client.use.data= node.hostname" to "true" on both the old and new cluster. Even though webhdfs is finally redirecting to data nodes using the public h= ostname, we keep seeing errors when running distcp. The errors are all simi= lar to: http://pastebin.com/ZYR07Fvm What do we need to do to get distcp to use the public hostname of the new m= achines? I haven't tried running distcp in the other direction (I'm about t= o), but I suspect I'll run into the same problem. Thanks! Jameel --_000_481e80d5e45f464ab991a9d5ec955cf96sensecom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi Ankit, 

We originally tried to copy to S3 and back. In fact, it is actually ou= r fallback plan. We were having issues with the copy to S3 not maintaining = the directory layout, so we decided to try and do a direct copy.

I'll give it another shot though!

Jameel Al-Aziz

Fro= m: Ankit Singhal <ankitsinghal59@gmail.com>
Sent: Sep 20, 2014 8:25 AM
To: user@hadoop.apache.org
Subject: Re: Unable to use transfer data using distcp between EC2-cl= assic cluster and VPC cluster

Hi Jameel,

As Peyman said, best approach is to do distcp from your old= cluster to s3 and making MR job reading directly from s3 on new cluster.

but If you still need to do distcp from hdfs to hdfs then update= /etc/hosts or DNS of all the nodes of your old cluster with "publicIp=   internalAWSDNSName" of all nodes of new cluster.
for eq:-
/etc/hosts of all nodes of old cluster should have entry of all = the nodes of new cluster in below format.
54.xxx.xxx.xx1   ip-10-xxx-xxx-xx1.ec2.internal=
54.xxx.xxx.xx2   ip-10-xxx-xxx-xx2.ec2.internal= =
54.xxx.xxx.xx3   ip-10-xxx-xxx-xx3.ec2.internal= =

Regards,
Ankit Singhal

On Sat, Sep 20, 2014 at 8:36 PM, Peyman Mohajeri= an <mohajeri@gmail.= com> wrote:
It maybe easier to copy the data to s3 and then from s3 to= the new cluster.

On Fri, Sep 19, 2014 at 8:45 PM, Jameel Al-Aziz = <jameel@6sense.co= m> wrote:
Hi all,

We’re in the process of migrating from EC2-Classic to VPC and needed = to transfer our HDFS data. We setup a new cluster inside the VPC, and assig= ned the name node and data node temporary public IPs. Initially, we had a l= ot of trouble getting the name node to redirect to the public hostname instead of private IPs. After some fiddlin= g around, we finally got webhdfs and dfs -cp to work using public hostnames= . However, distcp simply refuses to use the public hostnames when connectin= g to the data nodes.

We’re running distcp on the old cluster, copying data into the new cl= uster.

The old hadoop cluster is running 1.0.4 and the new one is running 1.2.1.

So far, on the new cluster, we’ve tried:
- Using public DNS hostnames in the master and slaves files (on both the na= me node and data nodes)
- Setting the hostname of all the boxes to their public DNS name
- Setting “fs.de= fault.name” to the public DNS name of the new name node.

And on both clusters:
- Setting the “dfs.datanode.use.datanode.hostname” and “d= fs.client.use.datanode.hostname” to “true" on both the old= and new cluster.

Even though webhdfs is finally redirecting to data nodes using the public h= ostname, we keep seeing errors when running distcp. The errors are all simi= lar to: htt= p://pastebin.com/ZYR07Fvm

What do we need to do to get distcp to use the public hostname of the new m= achines? I haven’t tried running distcp in the other direction (IR= 17;m about to), but I suspect I’ll run into the same problem.

Thanks!
Jameel


--_000_481e80d5e45f464ab991a9d5ec955cf96sensecom_--