Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <029C75A3482BE64594E21FC09BB19F411386C299@BY2PRD0711MB428.namprd07.prod.outlook.com>
References: 
 <CAMWNowFfkbCRw9BsiDJZHPFRtL0Vgj5xx+xbcH=AVU6w2qk=vw@mail.gmail.com>
 <029C75A3482BE64594E21FC09BB19F411386C299@BY2PRD0711MB428.namprd07.prod.outlook.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Fri, 12 Oct 2012 13:52:59 -0700
Message-ID: 
 <CAND0qzt1TH3zP7vnT9WBJuMk0xS4W7o-j2uKnGOt_3MD10FW1Q@mail.gmail.com>
Subject: Re: Spindle per Cores
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec54c52fcbb953904cbe2e05c

--bcaec54c52fcbb953904cbe2e05c
Content-Type: text/plain; charset=ISO-8859-1

I think that this rule of thumb is to prevent people configuring 2 disk
clusters with 16 cores or 48 disk machines with 4 cores.  Both
configurations could make sense in narrow applications, but both would most
probably be sub-optimal.

Within narrow bands, I doubt you will see huge changes.  I like to be able
to

a) be able to saturate disk I/O which requires some CPU and a good control.
 Different distros vary a lot here

b) have enough memory per slot.  Lots of people go cheap on this and they
wind up hamstringing performance

c) make sure there is enough CPU left over for the application.  This is
hugely app dependent, obviously.

On Fri, Oct 12, 2012 at 1:45 PM, Hank Cohen <hank.cohen@altior.com> wrote:

> What empirical evidence is there for this rule of thumb?
> In other words, what tests or metrics would indicate an optimal
> spindle/core ratio and how dependent is this on the nature of the data and
> of the map/reduce computation?
>
> My understanding is that there are lots of clusters with more spindles
> than cores.  Specifically, typical 2U servers can hold 12 3.5" disk drives.
>  So lots of Hadoop clusters have dual 4 core processors and 12 spindles.
>  Would it be better to have 6 core processors if you are loading up the
> boxes with 12 disks?  And most importantly, how would one know that the mix
> was optimal?
>
> Hank Cohen
> Altior Inc.
>
> -----Original Message-----
> From: Patai Sangbutsarakum [mailto:silvianhadoop@gmail.com]
> Sent: Friday, October 12, 2012 10:46 AM
> To: user@hadoop.apache.org
> Subject: Spindle per Cores
>
> I have read around about the hardware recommendation for hadoop cluster.
> One of them is recommend 1:1 ratio between spindle per core.
>
> Intel CPU come with Hyperthread which will double the number cores on one
> physical CPU. eg. 8 cores with Hyperthread it because 16 which is where we
> start to calculate about number of task slots per node.
>
> Once it come to spindle, i strongly believe I should pick 8 cores and
> picks 8 disks in order to get 1:1 ratio.
>
> Please suggest
> Patai
>
>
>

--bcaec54c52fcbb953904cbe2e05c
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I think that this rule of thumb is to prevent people configuring 2 disk clu=
sters with 16 cores or 48 disk machines with 4 cores. =A0Both configuration=
s could make sense in narrow applications, but both would most probably be =
sub-optimal.<div>

<br></div><div>Within narrow bands, I doubt you will see huge changes. =A0I=
 like to be able to=A0</div><div><br></div><div>a) be able to saturate disk=
 I/O which requires some CPU and a good control. =A0Different distros vary =
a lot here</div>

<div><br></div><div>b) have enough memory per slot. =A0Lots of people go ch=
eap on this and they wind up hamstringing performance</div><div><br></div><=
div>c) make sure there is enough CPU left over for the application. =A0This=
 is hugely app dependent, obviously.<br>

<br><div class=3D"gmail_quote">On Fri, Oct 12, 2012 at 1:45 PM, Hank Cohen =
<span dir=3D"ltr">&lt;<a href=3D"mailto:hank.cohen@altior.com" target=3D"_b=
lank">hank.cohen@altior.com</a>&gt;</span> wrote:<br><blockquote class=3D"g=
mail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l=
eft:1ex">

What empirical evidence is there for this rule of thumb?<br>
In other words, what tests or metrics would indicate an optimal spindle/cor=
e ratio and how dependent is this on the nature of the data and of the map/=
reduce computation?<br>
<br>
My understanding is that there are lots of clusters with more spindles than=
 cores. =A0Specifically, typical 2U servers can hold 12 3.5&quot; disk driv=
es. =A0So lots of Hadoop clusters have dual 4 core processors and 12 spindl=
es. =A0Would it be better to have 6 core processors if you are loading up t=
he boxes with 12 disks? =A0And most importantly, how would one know that th=
e mix was optimal?<br>


<br>
Hank Cohen<br>
Altior Inc.<br>
<br>
-----Original Message-----<br>
From: Patai Sangbutsarakum [mailto:<a href=3D"mailto:silvianhadoop@gmail.co=
m">silvianhadoop@gmail.com</a>]<br>
Sent: Friday, October 12, 2012 10:46 AM<br>
To: <a href=3D"mailto:user@hadoop.apache.org">user@hadoop.apache.org</a><br=
>
Subject: Spindle per Cores<br>
<br>
I have read around about the hardware recommendation for hadoop cluster.<br=
>
One of them is recommend 1:1 ratio between spindle per core.<br>
<br>
Intel CPU come with Hyperthread which will double the number cores on one p=
hysical CPU. eg. 8 cores with Hyperthread it because 16 which is where we s=
tart to calculate about number of task slots per node.<br>
<br>
Once it come to spindle, i strongly believe I should pick 8 cores and picks=
 8 disks in order to get 1:1 ratio.<br>
<br>
Please suggest<br>
Patai<br>
<br>
<br>
</blockquote></div><br></div>

--bcaec54c52fcbb953904cbe2e05c--