hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Armstrong <john.armstr...@ccri.com>
Subject Re: Passing a Global Variable into a Mapper
Date Thu, 15 Sep 2011 17:50:37 GMT
On Thu, 15 Sep 2011 12:43:57 -0500, Arko Provo Mukherjee
<arkoprovomukherjee@gmail.com> wrote:
> Is there a way to pass some data from the driver class to the Mapper
> class without going through the HDFS?

I generally use the Configuration object embedded in the Job for that.  My
Tool implements Configurable so I create by job with

    Job job = new Job(getConf(), "JobName");

Then I set variables that my job's tasks will need:

    job.getconfiguration().setString("property.name","property.value");

On the inside, usually in my Mapper.setup() or Reducer.setup() I extract
them:

    Configuration conf = context.getConfiguration();
    property = conf.get("property.name");

where property is a String field in my mapper that will hold the
property's value.  Of course, change things mutatis mutandis for storing
other kinds of values than Strings in the Configuration.

hth

Mime
View raw message