Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6ECA6ED20 for ; Wed, 27 Feb 2013 20:25:15 +0000 (UTC) Received: (qmail 69145 invoked by uid 500); 27 Feb 2013 20:25:10 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 68906 invoked by uid 500); 27 Feb 2013 20:25:10 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 68898 invoked by uid 99); 27 Feb 2013 20:25:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Feb 2013 20:25:10 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of viral.bajaria@gmail.com designates 209.85.219.48 as permitted sender) Received: from [209.85.219.48] (HELO mail-oa0-f48.google.com) (209.85.219.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Feb 2013 20:25:04 +0000 Received: by mail-oa0-f48.google.com with SMTP id j1so2110496oag.21 for ; Wed, 27 Feb 2013 12:24:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=K83FuoFAeqtIYOvEQ8KX1SFE6UiJXvmgSbgkogkKKsE=; b=Ykxajw1bmKCPuTRzvZg/L1rzYjOJmAhGJNGj2x0Gt1UZosuzfsS01glgmS1HBonya9 K0DGrUoB+dx79Ltax9eGpiFrLOaaBXTTE68SNMmBpvd+ydokbpjoqn1ZOFJNe7NwS8ky 3brRCGW6ZaAfY+nUNr6gUk0drf/ibSk41Wd0C+W4vmkHMN3bjUSYF4WC0wOM+8jH7qwP 3U1R+Mm/Zu9LUTuLfSTmim3vEaC1ctcHC5EyCO5h9jaNmVhsSVYIjbVuSgOSCn0YueRS Oi/QffYeB04R2nwIDFmz74hs3Wk0DwJlSJd2sn+l4+4oGa2smimkNIweByPZjBIKIsrl uo+A== MIME-Version: 1.0 X-Received: by 10.182.117.7 with SMTP id ka7mr3674201obb.29.1361996683014; Wed, 27 Feb 2013 12:24:43 -0800 (PST) Received: by 10.182.76.74 with HTTP; Wed, 27 Feb 2013 12:24:42 -0800 (PST) In-Reply-To: References: Date: Wed, 27 Feb 2013 12:24:42 -0800 Message-ID: Subject: Re: Correct way to unzip locally an archive in Yarn From: Viral Bajaria To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d0447878f79fdf604d6ba9011 X-Virus-Checked: Checked by ClamAV on apache.org --f46d0447878f79fdf604d6ba9011 Content-Type: text/plain; charset=ISO-8859-1 I was using 0.23 and was adding files using the -libjars flag (wanted to upload some jars which were dependencies for my project) but for some reason I could never find it in the DistributedCache or would always keep on getting ClassNotFound on the other side. I took the snippet of code which does that work when you invoke the hadoop jar command and put it in my class and it all worked fine. I don't know if the problem was with my code or if it was a bug. Given that -files is so extensively used, I felt it could be an issue on my side. In the end I started using 1.1 hadoop and so completely forgot about diving more deeper but I can definitely revive it and dig in more. Hopefully you can try using -libjars too and see if that also is facing a similar issue since they both are command line switches which should have almost similar behavior. Thanks, Viral On Tue, Feb 19, 2013 at 10:33 AM, Robert Evans wrote: > Yes if you can trace this down I would be very interested. We are running > 0.23.6 without any issues, but that does not mean that there is not some > bug in the code that is causing this to happen in your situation. > > --Bobby > > From: Sebastiano Vigna > Reply-To: "user@hadoop.apache.org" > Date: Saturday, February 16, 2013 8:39 AM > To: "user@hadoop.apache.org" > Subject: Re: Correct way to unzip locally an archive in Yarn > > I will as soon as I can understand what happens on the cluster (no access > from home). DistributedCache.getLocalCacheFiles() returns in both cases a > local name for the zip file uploaded with -files, but locally my unzip code > works, on the cluster it throws a FileNotFoundException. > > > On 16 February 2013 15:22, Arun C Murthy wrote: > >> This could be a bug, mind opening a jira? Thanks. >> >> On Feb 16, 2013, at 2:34 AM, Sebastiano Vigna wrote: >> >> On 15 February 2013 16:57, Robert Evans wrote: >> >>> Are you trying to run a Map/Reduce job or are you writing a new YARN >>> application? If it is a MR job, then it should work mostly the same as >>> before (on 1.x). If you are writing a new YARN application then there is >>> a >>> separate Map in the ContainerLaunchContext that you need to fill in. >> >> >> It's a MapReduce job (0.23.6). After two days of useless trials, I'm >> uploading the zip with -files and I wrote a stub to unzip it manually. I >> was positively unable to get the archive unzipped *to a local directory* in >> any way. >> >> Unfortunately it works in local but not on the cluster. I have still to >> discover why. :( >> >> Ciao, >> >> >> >> >> -- >> Arun C. Murthy >> Hortonworks Inc. >> http://hortonworks.com/ >> >> >> > --f46d0447878f79fdf604d6ba9011 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I was using 0.23 and was adding files using the -libjars flag (wanted = to upload some jars which were dependencies for my project) but for some re= ason I could never find it in the DistributedCache or would always keep on = getting ClassNotFound on the other side. I took the snippet of code which d= oes that work when you invoke the hadoop jar command and put it in my class= and it all worked fine. I don't know if the problem was with my code o= r if it was a bug. Given that -files is so extensively used, I felt it coul= d be an issue on my side. In the end I started using 1.1 hadoop and so comp= letely forgot about diving more deeper but I can definitely revive it and d= ig in more.

Hopefully you can try using -libjars too and see if tha= t also is facing a similar issue since they both are command line switches = which should have almost similar behavior.

Thanks,=
Viral

On Tue, Feb 19, 2013 at 10:33 = AM, Robert Evans <evans@yahoo-inc.com> wrote:
Yes if you can trace this down I would be very interested. =A0W= e are running 0.23.6 without any issues, but that does not mean that there = is not some bug in the code that is causing this to happen in your situatio= n.

--Bobby

From: Sebastiano Vigna <vigna@di.unimi.it>Reply-To: "user@hadoop.apache.org"= ; <user@hado= op.apache.org>
Date: Saturday, February 16, 2013 = 8:39 AM
To: "user@hadoop.apache.org&q= uot; <user@h= adoop.apache.org>
Subject: Re: Correct way to unzip = locally an archive in Yarn

<= div dir=3D"ltr">I will as soon as I can understand what happens on the clus= ter (no access from home). DistributedCache.getLocalCacheFiles= () returns in both cases a local name for the zip file uploaded with = -files, but locally my unzip code works, on the cluster it throws a FileNot= FoundException.


On 16 Februar= y 2013 15:22, Arun C Murthy <acm@hortonworks.com> wrote:
This could be a bug, mind opening a jir= a? Thanks.

On Feb 16, 2013, at 2:34 AM, Sebastiano Vigna = wrote:

On 15 Febru= ary 2013 16:57, Robert Evans=A0<evans@yahoo-inc.com>=A0wro= te:
Are you trying to run a Map/Reduce job or are you writing a new YARN
app= lication? =A0If it is a MR job, then it should work mostly the same as
b= efore (on 1.x). If you are writing a new YARN application then there is a separate Map in the ContainerLaunchContext that you need to fill in.
=A0
It's a MapReduce job=A0(0.23.6). After = two days of useless trials, I'm uploading the zip with -files and I wro= te a stub to unzip it manually. I was positively unable to get the archive = unzipped *to a local directory* in any way.

Unfortunately it works in local but not on the cluster. I ha= ve still to discover why. :(

Ciao,

<= div>


--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



--f46d0447878f79fdf604d6ba9011--