hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patai Sangbutsarakum <Patai.Sangbutsara...@turn.com>
Subject Re: Hardware Selection for Hadoop
Date Mon, 06 May 2013 22:57:56 GMT
I really doubt if he would spend $ to by 10 cores on a die CPU for "proof of concept" machines.
Actually, I even think of telling you to gathering old machines (but reliable) as much as
you can collect.
Put as much as disks, Ram you can. teaming up NIC if you can, and at that point you can proof
your concept up to certain point.

You will get the idea how is your application will behave, how big of the data set you will
play with
is the application cpu or io bound, and from that you can go out shopping buy the best fit
server configuration.

On May 6, 2013, at 4:17 AM, Michel Segel <michael_segel@hotmail.com<mailto:michael_segel@hotmail.com>>

8 physical cores is so 2009 - 2010 :-)

Intel now offers a chip w 10 physical cores on a die.
You are better off thinking of 4-8 GB per physical core.
It depends on what you want to do, and what you think you may want to do...

It also depends on the price points of the hardware. Memory, drives, CPUs (price by clock
speeds...) you just need to find the right optimum between price and performance...

Sent from a remote device. Please excuse any typos...

Mike Segel

On May 5, 2013, at 1:47 PM, Ted Dunning <tdunning@maprtech.com<mailto:tdunning@maprtech.com>>

Data nodes normally are also task nodes.  With 8 physical cores it isn't that unreasonable
to have 64GB whereas 24GB really is going to pinch.

Achieving highest performance requires that you match the capabilities of your nodes including
CPU, memory, disk and networking.  The standard wisdom is 4-6GB of RAM per core, at least
a spindle per core and 1/2 to 2/3 of disk bandwidth available as network bandwidth.

If you look at the different configurations mentioned in this thread, you will see different

For instance:

2 x Quad cores Intel
2-3 TB x 6 SATA         <==== 6 disk < desired 8 or more
64GB mem                <==== slightly larger than necessary
2 1GBe NICs teaming     <==== 2 x 100 MB << 400MB = 2/3 x 6 x 100MB

This configuration is mostly limited by networking bandwidth

2 x Quad cores Intel
2-3 TB x 6 SATA         <==== 6 disk < desired 8 or more
24GB mem                <==== 24GB << 8 x 6GB
2 10GBe NICs teaming    <==== 2 x 1000 MB > 400MB = 2/3 x 6 x 100MB

This configuration is weak on disk relative to CPU and very weak on disk relative to network
speed.  The worst problem, however, is likely to be small memory.  This will likely require
us to decrease the number of slots by half or more making it impossible to even use the 6
disks that we have and making the network even more outrageously over-provisioned.

On Sun, May 5, 2013 at 9:41 AM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com<mailto:rahul.rec.dgp@gmail.com>>
IMHO ,64 G looks bit high for DN. 24 should be good enough for DN.

On Tue, Apr 30, 2013 at 12:19 AM, Patai Sangbutsarakum <Patai.Sangbutsarakum@turn.com<mailto:Patai.Sangbutsarakum@turn.com>>
2 x Quad cores Intel
2-3 TB x 6 SATA
64GB mem
2 NICs teaming

my 2 cents

On Apr 29, 2013, at 9:24 AM, Raj Hadoop <hadoopraj@yahoo.com<mailto:hadoopraj@yahoo.com>>


I have to propose some hardware requirements in my company for a Proof of Concept with Hadoop.
I was reading Hadoop Operations and also saw Cloudera Website. But just wanted to know from
the group - what is the requirements if I have to plan for a 5 node cluster. I dont know at
this time, the data that need to be processed at this time for the Proof of Concept. So -
can you suggest something to me?


View raw message