hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: bringing the codebases back in line
Date Fri, 22 Oct 2010 11:37:17 GMT
On 21/10/10 22:53, Ian Holsman wrote:
> yep.. I've heard it's a source of contention...
> but I'd like to see how we can get it so the amount of patches that the
> large companies apply on top of the current production apache release gets
> minimized, and the large installations are all running nearly identical code
> on their clusters, and that we wouldn't need to have a yahoo or cloudera
> repo of their patch sets made available.
> So Ideally I'd like to hear what kind of things apache needs to do help get
> these kind of things less divergent.
> In discussing it with people, I've heard that a major issue (not the only
> one i'm sure) is lack of resources to actually test the apache releases on
> large clusters, and that it is very hard getting this done in short cycles
> (hence the large gap between 20.x and 21).
> So I thought I would start the thread to see if we could at least identify
> what the people think are the problems are.

A big issue is the $ value of the data in a production cluster, the size 
of the large clusters, and the fact that they are in use. The only way 
to test on a few hundred nodes -especially now that 12-24TB/node is 
possible, is when you are bringing up a cluster of this size, which is a 
pretty rare event. Lots of us have small real or virtual clusters, but 
they don't stress the NN or JT, don't raise emergent problems like the 
increasing cost of rebalancing 24TB nodes if they go down, etc.

View raw message