Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2C781D91B for ; Sat, 13 Oct 2012 03:28:28 +0000 (UTC) Received: (qmail 78092 invoked by uid 500); 13 Oct 2012 03:28:23 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 77596 invoked by uid 500); 13 Oct 2012 03:28:20 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 77557 invoked by uid 99); 13 Oct 2012 03:28:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Oct 2012 03:28:19 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ranjith.raghunath1@gmail.com designates 209.85.223.176 as permitted sender) Received: from [209.85.223.176] (HELO mail-ie0-f176.google.com) (209.85.223.176) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Oct 2012 03:28:11 +0000 Received: by mail-ie0-f176.google.com with SMTP id k11so6732063iea.35 for ; Fri, 12 Oct 2012 20:27:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=KreIEUj8va1Vr0GDHBB0/8JHC+s27fP/0Eyve9606AM=; b=wHBPmnUxsZ7TYH0af78Fg3zAZ1xHeIxRdS/az8u0oD52xhPFgQwy7ZIj41Lboz0D/6 qHY3U/gKmYW7Spqrb3/YBEoAp5ypWVXaG1UFAQ943Cxhv18tUiKb3kkt+X2foZ1lsWg4 7JDBd1P4lO3hBIjZ4s2N61y/8MVCYP9Y0ij2CZ/UYADioIBRSim44lbWgpw8Giq2UbFe EHjSU1mkkmty7+nHNdNo0tpOhHO1oiJ+oP73j2lHVENT8Dqq+IB5vUkg2CjpuVlyyf9z Htr0N2K6gTrXYVfxn8nivt0Yv60GtthaS74iOf5JpTlp6qj18B1J2o4vIEGPdCJIXCZP 9lxA== MIME-Version: 1.0 Received: by 10.50.237.6 with SMTP id uy6mr2745458igc.31.1350098870654; Fri, 12 Oct 2012 20:27:50 -0700 (PDT) Received: by 10.50.30.131 with HTTP; Fri, 12 Oct 2012 20:27:50 -0700 (PDT) Received: by 10.50.30.131 with HTTP; Fri, 12 Oct 2012 20:27:50 -0700 (PDT) In-Reply-To: References: <029C75A3482BE64594E21FC09BB19F411386C299@BY2PRD0711MB428.namprd07.prod.outlook.com> <71707C6AD2C02B4087F9E1BCA9EC816513C9E1A2C1@exch-mbx-112.vmware.com> Date: Fri, 12 Oct 2012 22:27:50 -0500 Message-ID: Subject: Re: Spindle per Cores From: ranjith raghunath To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d0447850d98d87204cbe863a4 X-Virus-Checked: Checked by ClamAV on apache.org --f46d0447850d98d87204cbe863a4 Content-Type: text/plain; charset=ISO-8859-1 Thanks Michael. On Oct 12, 2012 9:59 PM, "Michael Segel" wrote: > I think what we are seeing is the ratio based on physical Xeon cores. > So hyper threading wouldn't make any change to the actual ratio. > (1 disk per physical core, would be 1 disk per 2 virtual cores.) > > Again YMMV and of course thanks to this guy Moore who decided to write > some weird laws... the ratio could change over time as the CPUs become more > efficient and faster. > > > On Oct 12, 2012, at 9:52 PM, ranjith raghunath < > ranjith.raghunath1@gmail.com> wrote: > > Does hypertheading affect this ratio? > On Oct 12, 2012 9:36 PM, "Michael Segel" > wrote: > >> First, the obvious caveat... YMMV >> >> Having said that. >> >> The key here is to take a look across the various jobs that you will run. >> Some may be more CPU intensive, others more I/O intensive. >> >> If you monitor these jobs via Ganglia, when you have too few spindles you >> should see the wait cpu rise on the machines in the cluster. That is to >> say that you are putting an extra load on the systems because you're >> waiting for the disks to catch up. >> >> If you increase the ratio of disks to CPU, you should see that load drop >> as you are not wasting CPU cycles. >> >> Note that its not just the number of spindles, but also the bus and the >> controller cards that can also affect the throughput of disk I/O. >> >> Now just IMHO, there was a discussion on some of the CPU recommendations. >> To a point, it doesn't matter that much. You want to maximize the bang for >> the buck you can get w your hardware purchase. >> >> Use the ratio as a buying guide. Fewer than a ratio of 1 disk per core, >> and you're wasting the cpu that you bought. >> >> Going higher than a ratio of 1, like 1.5, and you may be buying too many >> spindles and not see a performance gain that offsets your cost. >> >> Search for a happy medium and don't sweat the maximum performance that >> you may get. >> >> HTH >> >> On Oct 12, 2012, at 4:19 PM, Jeffrey Buell wrote: >> >> > I've done some experiments along these lines. I'm using >> high-performance 15K RPM SAS drives instead of the more usual SATA drives, >> which should reduce the number of drives I need. I have dual 4-core >> processors at 3.6 GHz. These are more powerful than the average 4-core >> processor, which should increase the number of drives I need. Assuming >> these 2 effects cancel, then my results should also apply to machines with >> SATA drives and average processors. Using 8 drives (1-1) gets good >> performance for teragen and terasort. Going to 12 drives (1.5 per core) >> increases terasort performance by 15%. That might not seem like much >> compared to increasing the number of drives by 50%, but a better comparison >> is that 4 extra drives increased the cost of each machine by only about >> 12%, so the extra drives are (barely) worth it. If you're more time >> sensitive than cost sensitive, they they're definitely worth it. The extra >> drives did not help teragen, apparently because both CPU and the internal >> storage controller were close to saturation. So, of course everything >> depends on the app. You're shooting for saturated CPUs and disk bandwidth. >> Check that the CPU is not saturated (after checking Hadoop tuning and >> optimizing the number of tasks). Check that you have enough memory for more >> tasks with room leftover for a large buffer cache. Use 10 GbE networking >> or make sure the network has enough headroom. Check the storage controller >> can handle more bandwidth. If all are true (that is, no other >> bottlenecks), consider adding more drives. >> > >> > Jeff >> > >> >> -----Original Message----- >> >> From: Hank Cohen [mailto:hank.cohen@altior.com] >> >> Sent: Friday, October 12, 2012 1:46 PM >> >> To: user@hadoop.apache.org >> >> Subject: RE: Spindle per Cores >> >> >> >> What empirical evidence is there for this rule of thumb? >> >> In other words, what tests or metrics would indicate an optimal >> >> spindle/core ratio and how dependent is this on the nature of the data >> >> and of the map/reduce computation? >> >> >> >> My understanding is that there are lots of clusters with more spindles >> >> than cores. Specifically, typical 2U servers can hold 12 3.5" disk >> >> drives. So lots of Hadoop clusters have dual 4 core processors and 12 >> >> spindles. Would it be better to have 6 core processors if you are >> >> loading up the boxes with 12 disks? And most importantly, how would >> >> one know that the mix was optimal? >> >> >> >> Hank Cohen >> >> Altior Inc. >> >> >> >> -----Original Message----- >> >> From: Patai Sangbutsarakum [mailto:silvianhadoop@gmail.com] >> >> Sent: Friday, October 12, 2012 10:46 AM >> >> To: user@hadoop.apache.org >> >> Subject: Spindle per Cores >> >> >> >> I have read around about the hardware recommendation for hadoop >> >> cluster. >> >> One of them is recommend 1:1 ratio between spindle per core. >> >> >> >> Intel CPU come with Hyperthread which will double the number cores on >> >> one physical CPU. eg. 8 cores with Hyperthread it because 16 which is >> >> where we start to calculate about number of task slots per node. >> >> >> >> Once it come to spindle, i strongly believe I should pick 8 cores and >> >> picks 8 disks in order to get 1:1 ratio. >> >> >> >> Please suggest >> >> Patai >> >> >> > >> > >> >> > --f46d0447850d98d87204cbe863a4 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Thanks Michael.

On Oct 12, 2012 9:59 PM, "Michael Segel&quo= t; <michael_segel@hotmail.c= om> wrote:
I think what we are seeing is the ratio= based on physical Xeon cores.=A0
So hyper threading wouldn't make = any change to =A0the actual ratio.
(1 disk per physical core, wou= ld be 1 disk per 2 virtual cores.)=A0

Again YMMV and of course thanks to this guy Moore who d= ecided to write some weird laws... the ratio could change over time as the = CPUs become more efficient and faster.=A0


On Oct 12, 2012, at 9:52 PM, ranjith raghunath <ranjith.raghunat= h1@gmail.com> wrote:

Does hype= rtheading affect this ratio?

On Oct 12, 2012 9:36 PM, "Michael Segel&quo= t; <micha= el_segel@hotmail.com> wrote:
First, the obvious caveat... YMMV

Having said that.

The key here is to take a look across the various jobs that you will run. S= ome may be more CPU intensive, others more I/O intensive.

If you monitor these jobs via Ganglia, when you have too few spindles you s= hould see the wait cpu rise on the machines in the cluster. =A0That is to s= ay that you are putting an extra load on the systems because you're wai= ting for the disks to catch up.

If you increase the ratio of disks to CPU, you should see that load drop as= you are not wasting CPU cycles.

Note that its not just the number of spindles, but also the bus and the con= troller cards that can also affect the throughput of disk I/O.

Now just IMHO, there was a discussion on some of the CPU recommendations. T= o a point, it doesn't matter that much. You want to maximize the bang f= or the buck you can get w your hardware purchase.

Use the ratio as a buying guide. Fewer than a ratio of 1 disk per core, and= you're wasting the cpu that you bought.

Going higher than a ratio of 1, like 1.5, and you may be buying too many sp= indles and not see a performance gain that offsets your cost.

Search for a happy medium and don't sweat the maximum performance that = you may get.

HTH

On Oct 12, 2012, at 4:19 PM, Jeffrey Buell <jbuell@vmware.com> wrote:

> I've done some experiments along these lines. =A0I'm using hig= h-performance 15K RPM SAS drives instead of the more usual SATA drives, whi= ch should reduce the number of drives I need. =A0I have dual 4-core process= ors at 3.6 GHz. =A0These are more powerful than the average 4-core processo= r, which should increase the number of drives I need. =A0Assuming these 2 e= ffects cancel, then my results should also apply to machines with SATA driv= es and average processors. =A0Using 8 drives (1-1) gets good performance fo= r teragen and terasort. =A0Going to 12 drives (1.5 per core) increases tera= sort performance by 15%. =A0That might not seem like much compared to incre= asing the number of drives by 50%, but a better comparison is that 4 extra = drives increased the cost of each machine by only about 12%, so the extra d= rives are (barely) worth it. If you're more time sensitive than cost se= nsitive, they they're definitely worth it. =A0The extra drives did not = help teragen, apparently because both CPU and the internal storage controll= er were close to saturation. So, of course everything depends on the app. = =A0You're shooting for saturated CPUs and disk bandwidth. =A0Check that= the CPU is not saturated (after checking Hadoop tuning and optimizing the = number of tasks). Check that you have enough memory for more tasks with roo= m leftover for a large buffer cache. =A0Use 10 GbE networking or make sure = the network has enough headroom. =A0Check the storage controller can handle= more bandwidth. =A0If all are true (that is, no other bottlenecks), consid= er adding more drives.
>
> Jeff
>
>> -----Original Message-----
>> From: Hank Cohen [mailto:hank.cohen@altior.com]
>> Sent: Friday, October 12, 2012 1:46 PM
>> To: us= er@hadoop.apache.org
>> Subject: RE: Spindle per Cores
>>
>> What empirical evidence is there for this rule of thumb?
>> In other words, what tests or metrics would indicate an optimal >> spindle/core ratio and how dependent is this on the nature of the = data
>> and of the map/reduce computation?
>>
>> My understanding is that there are lots of clusters with more spin= dles
>> than cores. =A0Specifically, typical 2U servers can hold 12 3.5&qu= ot; disk
>> drives. =A0So lots of Hadoop clusters have dual 4 core processors = and 12
>> spindles. =A0Would it be better to have 6 core processors if you a= re
>> loading up the boxes with 12 disks? =A0And most importantly, how w= ould
>> one know that the mix was optimal?
>>
>> Hank Cohen
>> Altior Inc.
>>
>> -----Original Message-----
>> From: Patai Sangbutsarakum [mailto:silvianhadoop@gmail.com]
>> Sent: Friday, October 12, 2012 10:46 AM
>> To: us= er@hadoop.apache.org
>> Subject: Spindle per Cores
>>
>> I have read around about the hardware recommendation for hadoop >> cluster.
>> One of them is recommend 1:1 ratio between spindle per core.
>>
>> Intel CPU come with Hyperthread which will double the number cores= on
>> one physical CPU. eg. 8 cores with Hyperthread it because 16 which= is
>> where we start to calculate about number of task slots per node. >>
>> Once it come to spindle, i strongly believe I should pick 8 cores = and
>> picks 8 disks in order to get 1:1 ratio.
>>
>> Please suggest
>> Patai
>>
>
>


--f46d0447850d98d87204cbe863a4--