hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sujit Pal <sujit....@comcast.net>
Subject Re: how to pass values from mapper/reducer to driver code?
Date Thu, 24 Sep 2009 14:59:25 GMT
Thanks Anthony, that was what I was afraid it would be...

-sujit

On Wed, 2009-09-23 at 20:37 -0700, Anthony Urso wrote:
> Changing job configuration on a map process will only be visible on
> that process.
> 
> The expected communication vector between Mappers and Reducers is
> OutputCollector.collect().
> 
> Cheers,
> Anthony
> 
> On Wed, Sep 23, 2009 at 5:49 PM, Sujit Pal <sujit.pal@comcast.net> wrote:
> > Hi,
> >
> > I am using Hadoop 0.20. I can pass input values from the driver code
> > into the mapper/reducer code using the Configuration, something like
> > this:
> >
> > // driver code:
> > Job job = new Job(conf, "name");
> > job.getConfiguration("foo", "bar");
> >
> > // and in the mapper/reducer class, retrieve it like so:
> > String foo = context.getConfiguration().get("foo");
> >
> > I am calculating some values from within my mapper as a side effect.
> > Regardless of which mapper drops it, it is going to be the same (ie it
> > is not affected by the split size). So I tried doing something like
> > this:
> >
> > In the mapper's cleanup() method, I have:
> >
> > context.getConfiguration().set("r", "result");
> >
> > but when I try to retrieve it on the driver:
> > String result = job.getConfiguration().get("r");
> >
> > I get a null in the result, so it appears as though I am either using
> > the wrong data structure and that 2-way communication is not supported.
> >
> > Are there any other ways of getting at this sort of data? I read about
> > the Reporter, and I thought that perhaps I could use that, but not sure
> > how to do that.
> >
> > Any pointers would be greatly appreciated.
> >
> > TIA
> > Sujit
> >
> >
> >


Mime
View raw message