hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: Streamlining the Hadoop release process
Date Tue, 30 Apr 2013 17:39:02 GMT
We are running nightly tests on branch-2 and branch-0.23.  These include
unit and integration tests and branch-2 seems fairly good.  We would like
to move off of 0.23 to 2.0.  But, there are relatively few things in 2.0
that really push us in that direction over what we already have in 0.23.
The major ones are HDFS HA and RM recovery as steps towards rolling
upgrades.  For 2.0.5 to be beta for me really requires that the APIs are
locked down, for both wire and binary compatibility.  This fits with what
Arun has stated as his intentions and also with the compatibility
discussion that has been going on. All of this is great but also requires
a strong commitment from the entire community to really make this happen.
I wish I could have participated more in those discussions but with
everything else that has been on my plate I have not even had enough time
to read them in a timely fashion.  For us to adopt it widely we need 2.0.X
to be stable at scale.  We have worked out most of the issues on YARN,
there may be a few more because of the deltas between 0.23 and 2.0 but
that should be relatively small.  The big question is with HDFS.  The base
HA work seems to be fairly stable with the fuzz testing and everything
else that has happened, but I just don't know how it is going to handle a
4000 node cluster. I don't know if BigTop can help with that, but any type
of stress testing is desirable.  The other HDFS features like snapshots
just seem to add more risk for something that we do not see using in the
short term.  Not that it is a bad feature, we just don't need it yet.


On 4/29/13 11:34 PM, "Roman Shaposhnik" <rvs@apache.org> wrote:

>On Mon, Apr 29, 2013 at 11:53 AM, Robert Evans <evans@yahoo-inc.com>
>> This depends on how quickly Yahoo will be off of branch-0.23 and if
>> is the only one maintaining it then it will not receive any bug fixes.
>> Does that make branch-0.23 any less stable, no.  Does it make it less
>> desirable for people to move to? Yes.
>Do you think there's anything that we can do as a community to help with
>Or on a flip side -- do you think there's any chance for Yahoo! team
>to formalize
>the kind of workflows that the company cares about in Bigtop integration
>component unit tests?
>Personally, I'm mostly nervous about the fact that I'm yet to hear any of
>DevOps from major Hadoop using companies to chime in on the 2.0.5-beta
>discussion with their take on what 'beta' means to them.

View raw message