ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <o...@hortonworks.com>
Subject Re: Branch 0.0 or Trunk
Date Fri, 18 Nov 2011 14:25:48 GMT
On Fri, Nov 18, 2011 at 6:06 AM, Ahmed Fathalla <afathalla@gmail.com> wrote:

> But with the explosive growth of Hadoop, don't you think that we may be
> hitting this 4,500 machine limit soon? Shouldn't we design for scalability
> from the beginning?


Of course, although you need realistic expectations about how much you can
do at scale without testing it at scale. The trunk architecture should be
able to scale out to roughly 10,000 to 20,000 machines. If we need to scale
out further, we'll need a tree architecture. Zookeeper according to the
experts tops out at hundreds of writers.


> Eric mentioned that the data could be lost if
> controller crashes, how does trunk's architecture handle this case?


The definition and state of all of the clusters is stored in Zookeeper, so
no data is lost.

-- Owen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message