hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: running shell commands inside mapper
Date Wed, 14 Dec 2011 21:08:58 GMT

Yes and no.  Oozie does something like this, but not through the shell, and you can look at
how they are doing it.  I don't know the details, but you will probably need to get some delegation
tokens to make it work properly if you have security enabled.  You should be able to copy
the jar file over in the distributed cache without any problems.  The real question is why
are you doing this?  Oozie is doing it to avoid security problems with how it runs user code.
 Is this something to launch more jobs in some sort of a loop?  If so you need to be very
careful that you don't end up doing a DDOS on the JobTracker by submitting way too many jobs.

--Bobby Evans

On 12/14/11 5:24 AM, "souri datta" <souri.istheone@gmail.com> wrote:

 I am trying to run a shell command from within a mapper. The shell command is of the form:
  hadoop jar somjarfile arg1 arg2 ...

Can i do this type of operation from within a mapper?

Also, can i copy the jar file in distributed cache and use it?


View raw message