Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: neutral (herse.apache.org: local policy)
User-Agent: Microsoft-Entourage/11.3.3.061214
Date: Fri, 13 Jul 2007 09:27:00 -0700
Subject: Re: Client Scaling
From: Ted Dunning <tdunning@veoh.com>
To: <hadoop-user@lucene.apache.org>
Message-ID: <C2BCF5E4.18B74%tdunning@veoh.com>
Thread-Topic: Client Scaling
Thread-Index: AcfFaP1IO5AkJDFcEdy8vwANk7UzfgAAaYBG
In-Reply-To: <C2BCF320.2F9A1%marco@yahoo-inc.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

100 new files / 10 minutes =3D 0.17 files per second.

Any namenode at all should be able to handle this in its sleep.

100 clients x 10 MB / 10 minutes =3D 1GB / 600 seconds =3D 1.7 MB/s

This is actually very low aggregate bandwidth.  That many clients should be
able to handle this very easily.  Depending on what you need to do to the
data, you should be able to handle this with only 10 (possibly fewer).


On 7/13/07 9:15 AM, "Marco Nicosia" <marco@yahoo-inc.com> wrote:

> In general, I bet 100 clients should be no problem if you have a reasonab=
le
> number of dataNodes, especially if the client operations are not
> simultaneous.

>> How well does Hadoop scale for multiple client inputs? For instance, cou=
ld a
>> reasonably powerful namenode handle 100 client machines copying in 10 MB
>> every
>> 10 minutes? Assume all of the clients would be running a wrapper around =
the
>> "copyFromLocalFile" method.
>>=20
>> Thanks,
>>=20
>> Stu Hood
>> Webmail.us
>> "You manage your business. We'll manage your email."=AE