hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Bieniosek <mich...@powerset.com>
Subject Scaling hadoop up
Date Thu, 29 Mar 2007 20:21:39 GMT

When I try to scale Hadoop up to about 100 nodes on EC2 (single-cpu Xen), I
notice things start to fall apart.  For example, the jobtracker starts
dropping requests with the message "Call queue overflow discarding oldest
call".  I've also seen problems with the namenode where dfs requests fail
with EOFExceptions.

I've tried increasing the heartbeat value for the dfs (it's not configurable
for the jobtracker though).  Is there some other trick to make hadoop scale
a little further?  The website claims that Hadoop has scaled to 600 nodes,
but it seems like I would need a very powerful machine for the namenode and
jobtracker to do this.  Am I missing something?


View raw message