hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Viral Bajaria <viral.baja...@gmail.com>
Subject Re: Correct way to unzip locally an archive in Yarn
Date Wed, 27 Feb 2013 20:24:42 GMT
I was using 0.23 and was adding files using the -libjars flag (wanted to
upload some jars which were dependencies for my project) but for some
reason I could never find it in the DistributedCache or would always keep
on getting ClassNotFound on the other side. I took the snippet of code
which does that work when you invoke the hadoop jar command and put it in
my class and it all worked fine. I don't know if the problem was with my
code or if it was a bug. Given that -files is so extensively used, I felt
it could be an issue on my side. In the end I started using 1.1 hadoop and
so completely forgot about diving more deeper but I can definitely revive
it and dig in more.

Hopefully you can try using -libjars too and see if that also is facing a
similar issue since they both are command line switches which should have
almost similar behavior.


On Tue, Feb 19, 2013 at 10:33 AM, Robert Evans <evans@yahoo-inc.com> wrote:

> Yes if you can trace this down I would be very interested.  We are running
> 0.23.6 without any issues, but that does not mean that there is not some
> bug in the code that is causing this to happen in your situation.
> --Bobby
> From: Sebastiano Vigna <vigna@di.unimi.it>
> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Date: Saturday, February 16, 2013 8:39 AM
> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Subject: Re: Correct way to unzip locally an archive in Yarn
> I will as soon as I can understand what happens on the cluster (no access
> from home). DistributedCache.getLocalCacheFiles() returns in both cases a
> local name for the zip file uploaded with -files, but locally my unzip code
> works, on the cluster it throws a FileNotFoundException.
> On 16 February 2013 15:22, Arun C Murthy <acm@hortonworks.com> wrote:
>> This could be a bug, mind opening a jira? Thanks.
>> On Feb 16, 2013, at 2:34 AM, Sebastiano Vigna wrote:
>> On 15 February 2013 16:57, Robert Evans <evans@yahoo-inc.com> wrote:
>>> Are you trying to run a Map/Reduce job or are you writing a new YARN
>>> application?  If it is a MR job, then it should work mostly the same as
>>> before (on 1.x). If you are writing a new YARN application then there is
>>> a
>>> separate Map in the ContainerLaunchContext that you need to fill in.
>> It's a MapReduce job (0.23.6). After two days of useless trials, I'm
>> uploading the zip with -files and I wrote a stub to unzip it manually. I
>> was positively unable to get the archive unzipped *to a local directory* in
>> any way.
>> Unfortunately it works in local but not on the cluster. I have still to
>> discover why. :(
>> Ciao,
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/

View raw message