hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob Futrelle <bob.futre...@gmail.com>
Subject Re: hardware specs for hadoop nodes
Date Tue, 25 Sep 2007 16:27:36 GMT

How does Hadoop handle multi-core CPUs?  Does each core run a distinct copy
of the mapped app?  Is this automatic, or need some configuration, or what?  

I'm in the market to buy a few machines to set up a small cluster and am
wondering what I should consider.

Or should I just spread Hadoop over some friendly machines already in my
College, buying nothing?

 - Bob


Ted Dunning-3 wrote:
> 
> 
> We have an oddball collection of machines.  Most are in the class you
> mention, although some have single dual core CPU's and some have 12GB of
> memory.  We plan to use developer workstations at night (real soon now)
> which typically have 1-3GB of memory + single CPU.
> 
> Our name node is very lightly used because the files we analyze are pretty
> good sized (we produce only a few consolidated files per hour).
> 
> 
> On 9/10/07 4:54 PM, "John Heidemann" <johnh@isi.edu> wrote:
> 
>> 
>> What are reasonable hardware specifications for a Hadoop node?
>> Can we document this somewhere (maybe in the wiki as
>> HowToConfigureHardware?)
>> 
>> Obviously this will be a moving target, but some guidance about how much
>> CPU vs. memory vs. disk space is typical would be helpful.
>> 
>> As one datapoint, we are running some boxes that are 4 core, 64-bit @
>> 2GHz machines with 4GB of memory with [I think] 2 x 750GB disks.  I
>> think if I could I'd put 4 x 750GB disks in this box.  I believe this
>> configuration is basically the same as what came up in Yahoo!'s recent
>> sort benchmark.
>> 
>> Other datapoints anyone?
>> 
>> And what about, say on the namenode?  People talk about it being a
>> memory bottleneck, but ours is underutilized.
>> 
>> Should we start a wiki page about this?
>> 
>>    -John Heidemann
>> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/hardware-specs-for-hadoop-nodes-tf4419439.html#a12883435
Sent from the Hadoop Users mailing list archive at Nabble.com.


Mime
View raw message