hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Java Developer <javadev...@gmail.com>
Subject Can I wire a mapper and reducer class?
Date Thu, 12 Nov 2009 06:28:38 GMT
I want to know if I can "wire" mapper and reducer classes. By "wire" I mean
to use inversion of control (or dependency injection). For example, I see in
a lot of books and on the Map/Reduce tutorial page (
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html),
something like this:

conf.setMapperClass(MyMapper.class);
conf.setReducerClass(MyReducer.class);

However, I want do something like this:

MyMapper mapper = new MyMapper();
mapper.setSomeProperty("something");

MyReducer reducer = new MyReducer();
reducer.setSomeProperty("somethingElse");

JobConf conf = new JobConf();
conf.setMapper(mapper);
conf.setReducer(reducer);

Is this possible? I'm still learning MapReduce so I don't know if this is
possible. But all examples that I have come across follow the pattern on the
tutorial page. There is a work around that I've been forced to. In my mapper
and reducer class, I can grab some properties file and then fill in the
values that I need (for example for JDBC connection). I see the problem that
when I run this on a distributed environment, I'll need the properties files
there as well. Does the mapper and reducer classes have a reference back to
the JobConf object? If so, I suppose I can get the properties there.

Thanks.

Mime
View raw message