Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 01215F006 for ; Fri, 29 Mar 2013 14:57:46 +0000 (UTC) Received: (qmail 46333 invoked by uid 500); 29 Mar 2013 14:57:41 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 45929 invoked by uid 500); 29 Mar 2013 14:57:40 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 45896 invoked by uid 99); 29 Mar 2013 14:57:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Mar 2013 14:57:39 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of himanish@gmail.com designates 209.85.217.176 as permitted sender) Received: from [209.85.217.176] (HELO mail-lb0-f176.google.com) (209.85.217.176) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Mar 2013 14:57:33 +0000 Received: by mail-lb0-f176.google.com with SMTP id y8so382487lbh.21 for ; Fri, 29 Mar 2013 07:57:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=z2qz0+Rm5W6D3VOD3gBhmCktzB1XA9Wv5GEyTWuklUo=; b=NCO6ElKTtvNMGra6+hgWiBtNrdNZaJh9IIT4aoP3zTzgPdk8bz+AQ3fBHQ5/YSlGKD uVrEG1uL2TBAeHXngpLsOZ5BRuomFVSc8iFWJDB6v1l8aGlX+dJ3WweLLUU2M6lFzMKY +FyYecT1ZcGMlXlZ+7JMAgi7o11NV+jo07zkgHOSNByHZs/haKF6gIP7QLs99egebkiF juOEWcY/E7wcOaOpYdS2WoZ0Z2fU9trLb+F4MkkQZly0r2Lf9Axs2Akq6juJwLYn4IYc yMGiS9sKfUGjPl15Pq0Z7+0330x1B0XKISi4Buqso1kCwVLCFzLqr9/ieuOhJYN3FCdV ALFw== MIME-Version: 1.0 X-Received: by 10.112.39.106 with SMTP id o10mr1582006lbk.23.1364569032732; Fri, 29 Mar 2013 07:57:12 -0700 (PDT) Received: by 10.112.131.132 with HTTP; Fri, 29 Mar 2013 07:57:12 -0700 (PDT) In-Reply-To: References: Date: Fri, 29 Mar 2013 10:57:12 -0400 Message-ID: Subject: Re: Hadoop distcp from CDH4 to Amazon S3 - Improve Throughput From: Himanish Kushary To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=e0cb4efe2dac77c2fe04d9117c68 X-Virus-Checked: Checked by ClamAV on apache.org --e0cb4efe2dac77c2fe04d9117c68 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Yes you are right CDH4 is the 2.x line, but I even checked in the javadocs for 1.0.4 branch (could not find 1.0.3 API's so used http://hadoop.apache.org/docs/r1.0.4/api/index.html) but did not find the "ProgressableResettableBufferedFileInputStream" class.Not sure how it is present in the hadoop-core.jar in Amazon EMR. In the meantime I have come out with a dirty workaround by extracting the class from the Amazon jar and packaging it into its own separate jar.I am actually able to run the s3distcp now on local CDH4 using amazon's jar and transfer from my local hadoop to Amazon S3. But the real issue is the throughput. You mentioned that you had transferred 1.5 TB in 45 mins which comes to around 583 MB/s. I am barely getting 4 MB/s upload speed !! How did you get 100x times speed compared to me ? Could you please share any settings/tweaks that you may have done to achieve this. Were you on some very specific high bandwidth network ? Was is between HDFS on EC2 and amazon S3 ? Looking forward to hear from you. Thanks Himanish On Fri, Mar 29, 2013 at 10:34 AM, David Parks wrote= : > CDH4 can be either 1.x or2.x hadoop, are you using the 2.x line? I've use= d > it primarily with 1.0.3, which is what AWS uses, so I presume that's what > it's tested on. > > > Himanish Kushary wrote: > > Thanks Dave. > > I had already tried using the s3distcp jar. But got stuck on the below > error,which made me think that this is something specific to Amazon hadoo= p > distribution. > > Exception in thread "Thread-28" java.lang.NoClassDefFoundError: > org/apache/hadoop/fs/s3native/ProgressableResettableBufferedFileInputStre= am > > > Also, I noticed that the Amazon EMR hadoop-core.jar has this class but it > is not present on the CDH4 (my local env) hadoop jars. > > Could you suggest how I could get around this issue. One option could be > using the amazon specific jars but then probably I would need to get all > the jars ( else it could cause version mismatch errors for HDFS - > NoSuchMethodError etc etc ) > > Appreciate your help regarding this. > > - Himanish > > > > On Fri, Mar 29, 2013 at 1:41 AM, David Parks wrot= e: > >> None of that complexity, they distribute the jar publicly (not the >> source, but the jar). You can just add this to your libjars: s3n://* >> region*.elasticmapreduce/libs/s3distcp/*latest*/s3distcp.jar**** >> >> ** ** >> >> No VPN or anything, if you can access the internet you can get to S3. **= * >> * >> >> ** ** >> >> Follow their docs here: >> http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingE= MR_s3distcp.html >> **** >> >> ** ** >> >> Doesn=92t matter where you=92re Hadoop instance is running.**** >> >> ** ** >> >> Here=92s an example of code/parameters I used to run it from within anot= her >> Tool, it=92s a Tool, so it=92s actually designed to run from the Hadoop = command >> line normally.**** >> >> ** ** >> >> ToolRunner.*run*(getConf(), *new* S3DistCp(), *new* String[] {***= * >> >> "--src", "/frugg/image-cache-stage2/",**** >> >> "--srcPattern", ".*part.*",**** >> >> "--dest", "s3n://fruggmapreduce/results-"+env+ >> "/" + JobUtils.*isoDate* + "/output/itemtable/", **** >> >> "--s3Endpoint", "s3.amazonaws.com" });**** >> >> ** ** >> >> Watch the =93srcPattern=94, make sure you have that leading `.*`, that o= ne >> threw me for a loop once.**** >> >> ** ** >> >> Dave**** >> >> ** ** >> >> ** ** >> >> *From:* Himanish Kushary [mailto:himanish@gmail.com] >> *Sent:* Thursday, March 28, 2013 5:51 PM >> *To:* user@hadoop.apache.org >> *Subject:* Re: Hadoop distcp from CDH4 to Amazon S3 - Improve Throughput= * >> *** >> >> ** ** >> >> Hi Dave,**** >> >> ** ** >> >> Thanks for your reply. Our hadoop instance is inside our corporate >> LAN.Could you please provide some details on how i could use the s3distc= p >> from amazon to transfer data from our on-premises hadoop to amazon s3. >> Wouldn't some kind of VPN be needed between the Amazon EMR instance and = our >> on-premises hadoop instance ? Did you mean use the jar from amazon on ou= r >> local server ?**** >> >> ** ** >> >> Thanks**** >> >> On Thu, Mar 28, 2013 at 3:56 AM, David Parks >> wrote:**** >> >> Have you tried using s3distcp from amazon? I used it many times to >> transfer 1.5TB between S3 and Hadoop instances. The process took 45 min, >> well over the 10min timeout period you=92re running into a problem on.**= ** >> >> **** >> >> Dave**** >> >> **** >> >> **** >> >> *From:* Himanish Kushary [mailto:himanish@gmail.com] >> *Sent:* Thursday, March 28, 2013 10:54 AM >> *To:* user@hadoop.apache.org >> *Subject:* Hadoop distcp from CDH4 to Amazon S3 - Improve Throughput**** >> >> **** >> >> Hello,**** >> >> **** >> >> I am trying to transfer around 70 GB of files from HDFS to Amazon S3 >> using the distcp utility.There are aaround 2200 files distributed over 1= 5 >> directories.The max individual file size is approx 50 MB.**** >> >> **** >> >> The distcp mapreduce job keeps on failing with this error **** >> >> **** >> >> "Task attempt_201303211242_0260_m_000005_0 failed to report status for >> 600 seconds. Killing!" **** >> >> **** >> >> and in the task attempt logs I can see lot of INFO messages like **** >> >> **** >> >> "INFO org.apache.commons.httpclient.HttpMethodDirector: I/O exception >> (java.io.IOException) caught when processing request: Resetting to inval= id >> mark"**** >> >> **** >> >> I am thinking either transferring individual folders instead of the >> entire 70 GB folders as a workaround or as another option increasing the= " >> mapred.task.timeout" parameter to something like 6-7 hour ( as the avg >> rate of transfer to S3 seems to be 5 MB/s).Is there any other better >> option to increase the throughput for transferring bulk data from HDFS t= o >> S3 ? Looking forward for suggestions.**** >> >> **** >> >> **** >> >> -- >> Thanks & Regards >> Himanish **** >> >> >> >> **** >> >> ** ** >> >> -- >> Thanks & Regards >> Himanish **** >> > > > > -- > Thanks & Regards > Himanish > --=20 Thanks & Regards Himanish --e0cb4efe2dac77c2fe04d9117c68 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Yes you are right CDH4 is the 2.x line, but I even checked= in the javadocs for 1.0.4 branch (could=A0not find 1.0.3 API's so used= http://had= oop.apache.org/docs/r1.0.4/api/index.html) but did not find the "ProgressableResettableBufferedFileInputStream" class.Not sure how it = is present in the hadoop-core.jar in Amazon EMR.

In the meantime I have come out with a= dirty workaround by extracting the class from the Amazon jar and packaging= it into its own separate jar.I am actually able to run the s3distcp now on= local CDH4 using amazon's jar and transfer from my local hadoop to Ama= zon S3.

But t= he real issue is the throughput. You mentioned that you had transferred 1.5= TB in 45 mins which comes to around 583 MB/s. I am barely getting 4 MB/s u= pload speed !! How did you get 100x times speed compared to me ? Could you = please share any settings/tweaks that you may have done to=A0achieve=A0this= . Were you on some very specific high bandwidth network ? Was is between HD= FS on EC2 and amazon S3 ?

Looking forward to hear from you.
Thanks
Himanish


On Fri, Mar 29, 2013 at 10:34 AM, David Parks <davidparks21@yahoo.co= m> wrote:
CDH4 can be either 1.x or2.x hadoop, are you using the 2.x= line? I've used it primarily with 1.0.3, which is what AWS uses, so I = presume that's what it's tested on.


Himanish Kushary <himanish@gmail.com> wrote:

Thanks Dave.

I had already tried using the s3= distcp jar. But got stuck on the below error,which made me think that this = is something specific to Amazon hadoop distribution.

Exception in threa= d "Thread-28" java.lang.NoClassDefFoundError: org/apache/hadoop/f= s/s3native/ProgressableResettableBufferedFileInputStream=A0

Also, I noticed that the Amazon EMR hadoop-core.j= ar has this class but it is not present on the CDH4 (my local env) hadoop j= ars.

Could you suggest how I could get around this= issue. One option could be using the amazon specific jars but then probabl= y I would need to get all the jars ( else it could cause version mismatch e= rrors for HDFS - NoSuchMethodError etc etc )=A0

Appreciate your help regarding this.

- Himanish



On Fri, Mar 29, 2013 at 1:41 AM, David Parks <da= vidparks21@yahoo.com> wrote:
None of that complexity, they distribute the jar publicly (not the = source, but the jar). You can just add this to your libjars: s3n://region.elasticmapreduce/libs/s= 3distcp/latest/= s3distcp.jar

=A0<= u>

No VPN or anything, if you can access the inter= net you can get to S3.

=A0<= u>

Follow their docs here: http://docs.aws.amazon.com/ElasticMapReduce/latest/Dev= eloperGuide/UsingEMR_s3distcp.html

=A0

Doesn=92t matter where yo= u=92re Hadoop instance is running.

=A0

Here=92s an example of co= de/parameters I used to run it from within another Tool, it=92s a Tool, so = it=92s actually designed to run from the Hadoop command line normally.

=A0

=A0=A0=A0=A0=A0=A0 ToolRun= ner.run(getConf(), new S3DistCp(), new String[] {

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 "--src", =A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 "/frugg/image-cache-stage2/",

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 "--srcPattern&qu= ot;,=A0=A0=A0=A0= =A0 ".*part.*",=

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 "--dest", =A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 "s3n://fruggmapreduce/results-"+env+"/" + JobUtils.isoDate<= span style=3D"font-size:10pt;font-family:Consolas"> + "/output/ite= mtable/", <= /span>

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 "--s3Endpoint&qu= ot;,=A0=A0=A0=A0= =A0 "s3.amaz= onaws.com"=A0=A0=A0=A0=A0=A0=A0=A0 });

=A0

Watch the =93srcPattern= =94, make sure you have that leading `.*`, that one threw me for a loop onc= e.

=A0

Dave=

=A0

=A0<= /p>

From: Himan= ish Kushary [mailto:himanish@gmail.com]
Sent: Thursday, March 28, 2013 5:51 PM
To: user@hadoop.apache.org
= Subject: Re: Hadoop distcp from CDH4 to Amazon S3 - Improve Throughp= ut

=A0

Hi Dave,

<= /u>=A0

Thanks for your reply. Our hadoop instance i= s inside our corporate LAN.Could you please provide some details on how i c= ould use the s3distcp from amazon to transfer data from our on-premises had= oop to amazon s3. Wouldn't some kind of VPN be needed between the Amazo= n EMR instance and our on-premises hadoop instance ? Did you mean use the j= ar from amazon on our local server ?

=A0

Thanks

On Thu, Mar 28, 2013 at 3:56 AM, David P= arks <davidp= arks21@yahoo.com> wrote:

Have you tried using s3distcp from amazon? I used it m= any times to transfer 1.5TB between S3 and Hadoop instances. The process to= ok 45 min, well over the 10min timeout period you=92re running into a probl= em on.

=A0

Dave=

=A0

=A0<= /p>

From: Himan= ish Kushary [mailto:himanish@gmail.com]
Sent: Thursday, March 28, 2013 10:54 AM
To: user@hadoop.apache.orgSubject: Hadoop distcp from CDH4 to Amazon S3 - Improve Throughput<= /span>

=A0

Hello,<= /p>

=A0

I am trying to transfer around 70 GB of files from HDFS = to Amazon S3 using the distcp utility.There are aaround 2200 files distribu= ted over 15 directories.The max individual file size is approx 50 MB.

=A0

The distc= p mapreduce job keeps on failing with this error=A0

=A0

"Task attempt_201303211242_0260_m_000005_0 failed to report status f= or 600 seconds. Killing!" =A0

=A0

and in the= task attempt logs I can see lot of INFO messages like=A0<= /u>

=A0

"INFO org.apache.commons.httpclient.HttpMethodDirector: I/O exceptio= n (java.io.IOException) caught when processing request: Resetting to invali= d mark"

=A0

I am think= ing either transferring individual folders instead of the entire 70 GB fold= ers as a workaround or as another option increasing the=A0"mapre= d.task.timeout"=A0parameter to something like 6-7 hour ( a= s the avg rate of transfer to S3 seems to be 5 MB/s).Is the= re any other better option to increase the throughput for transferring bulk= data from HDFS to S3 ? =A0Looking forward for suggestions.

=A0

=A0

--
Thanks & Regards
Himanish

<= /div>



=A0

--
Thanks &= Regards
Himanish




--
Thanks & Regards
Himanish



--
= Thanks & Regards
Himanish
--e0cb4efe2dac77c2fe04d9117c68--