incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <josh.wi...@gmail.com>
Subject Re: Crunch Hadoop version support
Date Thu, 16 Aug 2012 00:00:43 GMT
In my mind, it's a function of how much surgery we would need to do in
order to support 0.20.2-- if it's just the multiple outputs stuff,
that seems like a small price to pay in terms of code duplication vs.
wider support. But I don't have a feel for how deep the rabbit hole
goes. If you wouldn't mind taking a crack at it and reporting back, we
would be much obliged-- but if you find yourself shaving a yak, you
should bail out. :)

J

On Wed, Aug 15, 2012 at 3:58 PM, Jonathan Dinu <jondinu@gmail.com> wrote:
> Hi,
>
> I have been using Crunch and starting to love it more and more.  I have been using it
with Hadoop-cdh3u4 and it has been swell but as Josh points out (http://mail-archives.apache.org/mod_mbox/incubator-crunch-user/201207.mbox/%3CCAH29n6N3GXkQ6B-NT3w3JwK8aMo7vDr9ip2=1xx0UF3KiKJ4ZA@mail.gmail.com%3E)
it depends on Hadoop being able to write multiple output files.  I have tried my code with
multiple versions of hadoop and it seems like 1.0.1 is the earliest version that I got my
tests to successful complete with.  0.20.x is not compatible because of the file output issue.
>
> I was curious if it is a goal of Crunch as it goes through Apache incubation to become
compatible with the 0.20.x versions of Hadoop (or more versions), if it is actually possible
to make it compatible with these versions, and if it is something that is desired by others.
>
> I have not yet contributed to Crunch, but I would be interested in helping to make it
compatible with the 0.20.x versions of Hadoop if the rest of the community thinks this is
a worthwhile endeavor.  Just wanted to get the opinion of the community and creators of Crunch.
>
> Best,
> Jonathan

Mime
View raw message