hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yang Chen" <chenyangyinp...@gmail.com>
Subject Re: MapWritable as output value of Reducer
Date Fri, 06 Jun 2008 00:14:07 GMT
I believe the (key, value) structure is same both input and output file. In
this case, you can consider the job flow.
Like below,
      JobConf confA = new JobConf(A.class);
     conf.setJobName("A");
     conf.setOutputKeyClass(Text.class);
     conf.setOutputValueClass(IntWritable.class);
     conf.setMapperClass(MapA.class);
     conf.setCombinerClass(ReduceA.class);
     conf.setReducerClass(ReduceA.class);
     conf.setInputFormat(TextInputFormat.class);
     conf.setOutputFormat(TextOutputFormat.class);
     conf.setInputPath(new Path("/inputA"));
     conf.setOutputPath(new Path("/outputA"));
     JobClient.runJob(confA);

      JobConf confB = new JobConf(B.class);
     conf.setJobName("B");
     conf.setOutputKeyClass(Text.class);
     conf.setOutputValueClass(IntWritable.class);
     conf.setMapperClass(MapB.class);
     conf.setCombinerClass(ReduceB.class);
     conf.setReducerClass(ReduceB.class);
     conf.setInputFormat(TextInputFormat.class);
     conf.setOutputFormat(TextOutputFormat.class);
     conf.setInputPath(new Path("/outputA"));
     conf.setOutputPath(new Path("/outputB"));
     JobClient.runJob(confB);

On Thu, Jun 5, 2008 at 7:36 PM, Tarandeep Singh <tarandeep@gmail.com> wrote:

> hi,
>
> Can I use MapWritable as an output value of a Reducer ?
>
> If yes, how will the (key, value) pairs in the MapWritable object will be
> written to the file ? What output format should I use in this case ?
>
> Further, I want to chain the output of the first map reduce job to another
> map reduce job, so in the second map reduce job, what input format should I
> specify ?
>
> Can I reconstruct the MapWritable objects in the mapper of the second job ?
>
> Thanks,
> Taran
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message