hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Update on hadoop-0.23
Date Tue, 27 Sep 2011 06:40:45 GMT

On Sep 26, 2011, at 11:20 PM, Arun C Murthy wrote:

> Roman, 
> In general, we'll need to make changes upstream: 
> # I believe someone got HBase working. 
> # We made changes to Pig - rather we got help from the Pig team, particularly Daniel.
> So, we plan to work through the rest of the stack - Hive, Oozie etc. very soon and we'll
depend on updated releases from the individual projects.

To clarify, the changes to Pig were mainly due to it's usage of the Context Objects apis which
have had changes in hadoop-0.21/hadoop-0.22.

Also, we expect some pieces of the stack to change if they rely on undocumented/hidden features
in MR.

We are absolutely committed to ensuring end-user MR applications have full compatibility -
to this end we have, long since, marked the old apis as stable & supported i.e. un-deprecated


> Arun
> On Sep 26, 2011, at 3:15 PM, Roman Shaposhnik wrote:
>> Hi Arun!
>> Great news! Hopefuly you wouldn't mind answering some of the questions below...
>> On Mon, Sep 26, 2011 at 2:07 PM, Arun C Murthy <acm@hortonworks.com> wrote:
>>> NextGen MapReduce (aka MRv2, aka YARN) is coming along great:
>>> # We are happy to report we've done extensive scale testing to confirm stability
>>> - Sort/GridMixv3 etc. at ~350nodes
>>> - Scale testing with simulated clusters of ~1500 nodes
>>> # Functional tests for all of MapReduce functionality
>>> # Pig  (0.9 & 0.9.1) working with NextGen MapReduce
>> Is there a *released* version of Pig that compiles cleanly against .23
>> snapshots?
>> Same question for Hive.
>>> We are about to finish performance certification for both HDFS & MapReduce
in the next
>>> couple of weeks too, after which we start integration tests with HBase, Hive,
Oozie etc.
>> I'm curious -- what are these integrations tests? Can I take a look at
>> them? I would
>> be really nice if we can levarage those via Bigtop infrastructure. Currently we
>> have a certain # of integration tests in Bigtop that we're running
>> against a fully
>> deployed stack, but it would be quite nice to have extra coverage.
>>> Given where we are I'm confident we can have a strong hadoop-0.23.0 release
>>> by late October. The current plan is to deploy to alpha clusters in November.
Citius, Altius, Fortius! :)
>> Could you, please, elaborate on what will be part of that deployment?
>> Which versions
>> of Pig, Hive, HBase, Oozie and Mahout are you targeting?
>> Thanks,
>> Roman.

View raw message