hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: combiner/reducer context in java class
Date Tue, 06 Nov 2012 11:29:36 GMT
Hi Prasad,

My reply inline.

On Tue, Nov 6, 2012 at 4:15 PM, Prasad GS <gsp200183@gmail.com> wrote:
> Hi,
> I'm setting my combiner and reducer to the same java class. Is there any API
> that could tell me the context in which the java class is running after the
> hadoop job is submitted to the cluster i.e whether the class is running as a
> combiner or a reducer.

A combiner may run both at the map end and at the reduce end. Even if
it is possible to do it, it isn't a healthy idea to have the method's
logic detect if its running as a reducer or as a combiner.

> I need this information to change the OutputCollector
> in the java class. Also I do not want to duplicate the same code as combiner
> and reducer with only the OutputCollector changed.

Why do you think it would require duplication? Your logic can be built
in smaller, independent, reusable functions within the same class, and
just applied differently for an implementation of Reducer class and an
implementation of the Combiner class. This way, you repeat nothing.

> Thanks,
> Prasad

Harsh J

View raw message