hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kumar Harshit <hkumar.ar...@gmail.com>
Subject Re: Running jar files inside map task
Date Wed, 27 Oct 2010 09:16:08 GMT
I think this is not difficult. Let me give an example here.

There are two Map Reduce job, MR1 and MR2.
MR1 has Map1 and Reduce1 as mapper and reducer.
MR2 has Map2 and Reduce2 as mapper and reducer.

MR1 input directory is Input1 and output directory is Output1
MR2 input directory is Output1 (*this makes sure that Output of MR1 is input
to MR2*) and output directory is MR2.

In the main() function of MR1, specify the paths of input directory and
output directory:
setInputPaths(new Path("Input1"));
setOutputPaths(new Path("Output1"));

In the main() function of MR2, specify the path of input directory and
output directory
*setInputPaths(new Path("Output1")); //Note that, the Output directory of
Previous MR1 is now the input directory o*
setOutputPaths(new Path("Output2"));

Try to search on net for more examples. there are plenty of them.

Best
Kumar
On Wed, Oct 27, 2010 at 3:22 AM, gaurav bagga <gaur.vbagga@gmail.com> wrote:

> It would be great if you could tell or point me to an article which uses
> the
> output of first map reduce as input for the 2nd map reduce.
>
> -Gaurav
>
>
>
> On Tue, Oct 26, 2010 at 7:02 PM, Kumar Harshit <hkumar.arora@gmail.com
> >wrote:
>
> > You can create 2nd map reduce job. The input to the mapper of 2nd Map
> > Reduce
> > job is the output of 1st Map Reduce job. This way you can tackle the
> issue.
> >
> > Hope it helps
> >
> > Kumar
> >
> > On Mon, Oct 25, 2010 at 1:42 PM, Ankit Gandhi <ankit.g1290@gmail.com>
> > wrote:
> >
> > > Hey,
> > > I want to know whether can I run a jar file inside a map task or not
> > > because
> > > I have to use the output of that file in my map task.
> > > I am able to run it in standalone mode but it fails in
> psuedo-distributed
> > > mode.
> > > Thanks in advance
> > >
> > > --
> > > Ankit Gandhi
> > > Undergraduate
> > > Center for Visual Information Technology
> > > Computer Science Engineering & Dual Degree
> > > IIIT-Hyderabad
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message