incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Friedrich <>
Subject Re: Supporting legacy Mapper and Reducer classes in Crunch
Date Tue, 25 Sep 2012 17:47:20 GMT
On Monday, 2012-09-24, Josh Wills wrote:
> This turns out to be kind of tricky to do no matter how we approach
> the problem, because for this to work, we'll need to (at a minimum)
> subclass the Mapper.Context and Reducer.Context classes that are
> passed to the Mapper and Reducer instances, and they have different
> implementations (most importantly for our purposes, different
> constructors) under Hadoop 1 and 2.
> It feels to me that what I need to do is create a separate subproject
> that has to do some crazy stuff (e.g., use different source
> directories depending on the value of the crunch.platform variable) in
> order to be able to create the appropriate kind of subclass of
> Mapper.Context or Reducer.Context. But this sort of thing seems like
> such a bad idea that there must be some sort of less-bad option
> available to me, and I wanted to solicit input before I start tilting
> at this particular windmill.

I haven't looked at the problem in detail, but perhaps you can get
away with a bit of reflection. Get the class object of Mapper.Context,
get its Constructors to figure out if you're running Hadoop 1 or 2,
and then use Constructor.newInstance() with the right argument list
depending on the version of Hadoop.

This doesn't feel a lot better than the Maven-based solution, but at
least you can encapsulate it deep in a class somewhere ;-)


View raw message