hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: [DISCUSSION] Thinking about 20.204 and beyond
Date Wed, 22 Jun 2011 09:30:15 GMT
On 21/06/2011 19:40, Roy T. Fielding wrote:
> On Jun 21, 2011, at 4:39 AM, Steve Loughran wrote:
>> On 18/06/2011 21:22, Roy T. Fielding wrote:
>> olutely no reason that trunk cannot be packaged for release
>>> tomorrow as 0.23.  There may be many reasons why it won't pass a release
>>> vote, but we probably aren't going to find them until somebody tries.
>> One limitation with releases has always been size of cluster testing -where Yahoo!s
contributions have been invaluable. That said, we shouldn't make them an SPOF in the release
process; we should all set up to do some more release testing.
> Yes, more testing is better, but if it can't be tested by the dev team
> in 72 hours then it doesn't belong in our release process.
> Please note that one of the main advantages of open source development
> is that the bulk of testing/QA occurs *after* the release.  That's why
> labels like alpha/beta/GA are best applied/updated after the version number
> has been cut and the software has been proven in real deployments.
> If testing on 5000 nodes is important to our customers, then add a
> scale-tested metric to the download site so that the customers know
> which release package has been tested at what scale -- they will
> understand the difference between frequent releases and those fully tested
> at scale.  Let them decide which version is best to use for their own needs.

I agree with in-field testing; the big issue there is that people who do 
have 1+PB of data are nervous about Hadoop upgrades, JVM upgrades. I 
haven't even heard of anyone who owns up to moving to ext4 fs 
underneath. It's the cost of loss of data that raises concerns, not the 
loss of time in testing and rolling back [1]. And very few people have 
500+ node clusters sitting around idle.

I like the point about mixed endian; Sparc may be dead but there are 
other architectures out there, and Arm looks up and coming. Then there 
are bits of the config space like a block replication factor of 2 that 
can be tested at small scale.


[1] Steve reverted his work desktop from RHEL6 to Ubuntu 10.04 last week 
and, after discussion with his 9 year old sun, is going to switch back 
to Firefox 3.6 as it does better flash games)

View raw message