hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: Client Scaling
Date Fri, 13 Jul 2007 16:27:00 GMT
100 new files / 10 minutes = 0.17 files per second.

Any namenode at all should be able to handle this in its sleep.

100 clients x 10 MB / 10 minutes = 1GB / 600 seconds = 1.7 MB/s

This is actually very low aggregate bandwidth.  That many clients should be
able to handle this very easily.  Depending on what you need to do to the
data, you should be able to handle this with only 10 (possibly fewer).


On 7/13/07 9:15 AM, "Marco Nicosia" <marco@yahoo-inc.com> wrote:

> In general, I bet 100 clients should be no problem if you have a reasonable
> number of dataNodes, especially if the client operations are not
> simultaneous.

>> How well does Hadoop scale for multiple client inputs? For instance, could a
>> reasonably powerful namenode handle 100 client machines copying in 10 MB
>> every
>> 10 minutes? Assume all of the clients would be running a wrapper around the
>> "copyFromLocalFile" method.
>> 
>> Thanks,
>> 
>> Stu Hood
>> Webmail.us
>> "You manage your business. We'll manage your email."®


Mime
View raw message