hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Update on hadoop-0.23
Date Tue, 27 Sep 2011 06:40:45 GMT

On Sep 26, 2011, at 11:20 PM, Arun C Murthy wrote:

> Roman, 
> 
> In general, we'll need to make changes upstream: 
> # I believe someone got HBase working. 
> # We made changes to Pig - rather we got help from the Pig team, particularly Daniel.
> 
> So, we plan to work through the rest of the stack - Hive, Oozie etc. very soon and we'll
depend on updated releases from the individual projects.
> 

To clarify, the changes to Pig were mainly due to it's usage of the Context Objects apis which
have had changes in hadoop-0.21/hadoop-0.22.

Also, we expect some pieces of the stack to change if they rely on undocumented/hidden features
in MR.

We are absolutely committed to ensuring end-user MR applications have full compatibility -
to this end we have, long since, marked the old apis as stable & supported i.e. un-deprecated
them.

Arun

> Arun
> 
> On Sep 26, 2011, at 3:15 PM, Roman Shaposhnik wrote:
> 
>> Hi Arun!
>> 
>> Great news! Hopefuly you wouldn't mind answering some of the questions below...
>> 
>> On Mon, Sep 26, 2011 at 2:07 PM, Arun C Murthy <acm@hortonworks.com> wrote:
>>> NextGen MapReduce (aka MRv2, aka YARN) is coming along great:
>>> # We are happy to report we've done extensive scale testing to confirm stability
>>> - Sort/GridMixv3 etc. at ~350nodes
>>> - Scale testing with simulated clusters of ~1500 nodes
>>> # Functional tests for all of MapReduce functionality
>>> # Pig  (0.9 & 0.9.1) working with NextGen MapReduce
>> 
>> Is there a *released* version of Pig that compiles cleanly against .23
>> snapshots?
>> Same question for Hive.
>> 
>>> We are about to finish performance certification for both HDFS & MapReduce
in the next
>>> couple of weeks too, after which we start integration tests with HBase, Hive,
Oozie etc.
>> 
>> I'm curious -- what are these integrations tests? Can I take a look at
>> them? I would
>> be really nice if we can levarage those via Bigtop infrastructure. Currently we
>> have a certain # of integration tests in Bigtop that we're running
>> against a fully
>> deployed stack, but it would be quite nice to have extra coverage.
>> 
>>> Given where we are I'm confident we can have a strong hadoop-0.23.0 release
>>> by late October. The current plan is to deploy to alpha clusters in November.
Citius, Altius, Fortius! :)
>> 
>> Could you, please, elaborate on what will be part of that deployment?
>> Which versions
>> of Pig, Hive, HBase, Oozie and Mahout are you targeting?
>> 
>> Thanks,
>> Roman.
> 


Mime
View raw message