hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Urso <anthony.u...@gmail.com>
Subject Re: how to pass values from mapper/reducer to driver code?
Date Thu, 24 Sep 2009 03:37:52 GMT
Changing job configuration on a map process will only be visible on
that process.

The expected communication vector between Mappers and Reducers is
OutputCollector.collect().

Cheers,
Anthony

On Wed, Sep 23, 2009 at 5:49 PM, Sujit Pal <sujit.pal@comcast.net> wrote:
> Hi,
>
> I am using Hadoop 0.20. I can pass input values from the driver code
> into the mapper/reducer code using the Configuration, something like
> this:
>
> // driver code:
> Job job = new Job(conf, "name");
> job.getConfiguration("foo", "bar");
>
> // and in the mapper/reducer class, retrieve it like so:
> String foo = context.getConfiguration().get("foo");
>
> I am calculating some values from within my mapper as a side effect.
> Regardless of which mapper drops it, it is going to be the same (ie it
> is not affected by the split size). So I tried doing something like
> this:
>
> In the mapper's cleanup() method, I have:
>
> context.getConfiguration().set("r", "result");
>
> but when I try to retrieve it on the driver:
> String result = job.getConfiguration().get("r");
>
> I get a null in the result, so it appears as though I am either using
> the wrong data structure and that 2-way communication is not supported.
>
> Are there any other ways of getting at this sort of data? I read about
> the Reporter, and I thought that perhaps I could use that, but not sure
> how to do that.
>
> Any pointers would be greatly appreciated.
>
> TIA
> Sujit
>
>
>

Mime
View raw message