hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: hardware specs for hadoop nodes
Date Tue, 11 Sep 2007 00:04:13 GMT

We have an oddball collection of machines.  Most are in the class you
mention, although some have single dual core CPU's and some have 12GB of
memory.  We plan to use developer workstations at night (real soon now)
which typically have 1-3GB of memory + single CPU.

Our name node is very lightly used because the files we analyze are pretty
good sized (we produce only a few consolidated files per hour).

On 9/10/07 4:54 PM, "John Heidemann" <johnh@isi.edu> wrote:

> What are reasonable hardware specifications for a Hadoop node?
> Can we document this somewhere (maybe in the wiki as HowToConfigureHardware?)
> Obviously this will be a moving target, but some guidance about how much
> CPU vs. memory vs. disk space is typical would be helpful.
> As one datapoint, we are running some boxes that are 4 core, 64-bit @
> 2GHz machines with 4GB of memory with [I think] 2 x 750GB disks.  I
> think if I could I'd put 4 x 750GB disks in this box.  I believe this
> configuration is basically the same as what came up in Yahoo!'s recent
> sort benchmark.
> Other datapoints anyone?
> And what about, say on the namenode?  People talk about it being a
> memory bottleneck, but ours is underutilized.
> Should we start a wiki page about this?
>    -John Heidemann

View raw message