hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <michael_se...@hotmail.com>
Subject RE: Running jar files inside map task
Date Wed, 27 Oct 2010 14:20:21 GMT

Uhm... just to take this in a different direction ...

Yes, you can run a jar file within your hadoop m/r job. 
You may end up with class loader problems though, depending on what's in the jar.

I think what I'm talking about goes beyond the concept of chaining jobs, so I'm just tossing
it out there as a response to your question ...

HTH

-Mike


> From: qwertymaniac@gmail.com
> Date: Wed, 27 Oct 2010 15:21:28 +0530
> Subject: Re: Running jar files inside map task
> To: general@hadoop.apache.org
> 
> On Wed, Oct 27, 2010 at 12:52 PM, gaurav bagga <gaur.vbagga@gmail.com> wrote:
> > It would be great if you could tell or point me to an article which uses the
> > output of first map reduce as input for the 2nd map reduce.
> 
> There's ChainMapper and ChainReducer that let you do [MAP+ / REDUCE
> MAP*] kind of job configuration (single job).
> 
> If you wish to chain jobs (To look like MRMRMR-ish), look at
> o.a.h.mapred.jobcontrol.Job at:
> http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred/jobcontrol/Job.html
> 
> Specifically, look at Job.addDependingJob.
> 
> >
> > -Gaurav
> >
> >
> >
> > On Tue, Oct 26, 2010 at 7:02 PM, Kumar Harshit <hkumar.arora@gmail.com>wrote:
> >
> >> You can create 2nd map reduce job. The input to the mapper of 2nd Map
> >> Reduce
> >> job is the output of 1st Map Reduce job. This way you can tackle the issue.
> >>
> >> Hope it helps
> >>
> >> Kumar
> >>
> >> On Mon, Oct 25, 2010 at 1:42 PM, Ankit Gandhi <ankit.g1290@gmail.com>
> >> wrote:
> >>
> >> > Hey,
> >> > I want to know whether can I run a jar file inside a map task or not
> >> > because
> >> > I have to use the output of that file in my map task.
> >> > I am able to run it in standalone mode but it fails in psuedo-distributed
> >> > mode.
> >> > Thanks in advance
> >> >
> >> > --
> >> > Ankit Gandhi
> >> > Undergraduate
> >> > Center for Visual Information Technology
> >> > Computer Science Engineering & Dual Degree
> >> > IIIT-Hyderabad
> >> >
> >>
> >
> 
> 
> 
> -- 
> Harsh J
> www.harshj.com
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message