crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-235) Avoid exposing incompatible Hadoop classes in Crunch API
Date Fri, 05 Jul 2013 15:03:48 GMT


Josh Wills commented on CRUNCH-235:

Hey Tom-- I'd be all for this if it really meant that Crunch apps could swap in the hadoop-2
APIs without a recompile, but I don't think I've ever been able to do that, even against apps
that didn't use Counters at all. The MR execution code depends too closely on the TaskAttemptContext-and-friends
stuff that went from class to interface to ever let people run against hadoop2 w/o a recompile--
unless there's some trick to doing it that I haven't figured out.
> Avoid exposing incompatible Hadoop classes in Crunch API
> --------------------------------------------------------
>                 Key: CRUNCH-235
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: CRUNCH-235.patch
> Between Hadoop 1 and 2, org.apache.hadoop.mapreduce.Counter changed from a class to an
interface. Therefore, exposing Counter in Crunch's API means that Crunch programs may need
to be recompiled when moving from a Hadoop 1 to a Hadoop 2 cluster. It would be nice to avoid
the need to recompile. 
> Note that Crunch itself has two artifacts - one for each major version of Hadoop - and
the change proposed here would not alter that, it would just mean that the same Crunch program
binary could be used with either Crunch artifact.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message