Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 873DADD6F for ; Sat, 13 Oct 2012 02:53:12 +0000 (UTC) Received: (qmail 10198 invoked by uid 500); 13 Oct 2012 02:53:07 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 10074 invoked by uid 500); 13 Oct 2012 02:53:07 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 10066 invoked by uid 99); 13 Oct 2012 02:53:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Oct 2012 02:53:07 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ranjith.raghunath1@gmail.com designates 209.85.210.176 as permitted sender) Received: from [209.85.210.176] (HELO mail-ia0-f176.google.com) (209.85.210.176) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Oct 2012 02:53:00 +0000 Received: by mail-ia0-f176.google.com with SMTP id h11so2877374iae.35 for ; Fri, 12 Oct 2012 19:52:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=OQwEReqWEp7OZAsYHMg806C+5VT0FI8qtDl7aJE5aoY=; b=RzP5uksCh/VgqtIRZqZt597lIzz92njH9VeFEFeWaUEuKbBMdRrSIEsrFfUpsiVGlZ f4ICYzGgXhT8AD7YLM1iTMJ83JsbVMCuDhsECEbETRXV4Hce/E1d0niOGJ0hmPULbqnk h5kH5iPoHt6/1TABWPNwxLViNJokn1bonTqdwZM7NVAqmUNiVoKeW09awbTwT0iQEtYP CmZfdaxjWjlQYUo/S4YGVx7OBO7YsVxXlzoypiObVuWwk28LtZZeKnZE5GovLym5POe/ DJzvc+Q1RXxzFbaZIBXsaE3LRFvBsOd3LEwPybYRMutNVrqkoxMh0GVXaaxx6CfCrjaA NnDA== MIME-Version: 1.0 Received: by 10.50.179.33 with SMTP id dd1mr3940306igc.31.1350096759305; Fri, 12 Oct 2012 19:52:39 -0700 (PDT) Received: by 10.50.30.131 with HTTP; Fri, 12 Oct 2012 19:52:39 -0700 (PDT) Received: by 10.50.30.131 with HTTP; Fri, 12 Oct 2012 19:52:39 -0700 (PDT) In-Reply-To: References: <029C75A3482BE64594E21FC09BB19F411386C299@BY2PRD0711MB428.namprd07.prod.outlook.com> <71707C6AD2C02B4087F9E1BCA9EC816513C9E1A2C1@exch-mbx-112.vmware.com> Date: Fri, 12 Oct 2012 21:52:39 -0500 Message-ID: Subject: Re: Spindle per Cores From: ranjith raghunath To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d0447a10fc0364704cbe7e559 X-Virus-Checked: Checked by ClamAV on apache.org --f46d0447a10fc0364704cbe7e559 Content-Type: text/plain; charset=ISO-8859-1 Does hypertheading affect this ratio? On Oct 12, 2012 9:36 PM, "Michael Segel" wrote: > First, the obvious caveat... YMMV > > Having said that. > > The key here is to take a look across the various jobs that you will run. > Some may be more CPU intensive, others more I/O intensive. > > If you monitor these jobs via Ganglia, when you have too few spindles you > should see the wait cpu rise on the machines in the cluster. That is to > say that you are putting an extra load on the systems because you're > waiting for the disks to catch up. > > If you increase the ratio of disks to CPU, you should see that load drop > as you are not wasting CPU cycles. > > Note that its not just the number of spindles, but also the bus and the > controller cards that can also affect the throughput of disk I/O. > > Now just IMHO, there was a discussion on some of the CPU recommendations. > To a point, it doesn't matter that much. You want to maximize the bang for > the buck you can get w your hardware purchase. > > Use the ratio as a buying guide. Fewer than a ratio of 1 disk per core, > and you're wasting the cpu that you bought. > > Going higher than a ratio of 1, like 1.5, and you may be buying too many > spindles and not see a performance gain that offsets your cost. > > Search for a happy medium and don't sweat the maximum performance that you > may get. > > HTH > > On Oct 12, 2012, at 4:19 PM, Jeffrey Buell wrote: > > > I've done some experiments along these lines. I'm using > high-performance 15K RPM SAS drives instead of the more usual SATA drives, > which should reduce the number of drives I need. I have dual 4-core > processors at 3.6 GHz. These are more powerful than the average 4-core > processor, which should increase the number of drives I need. Assuming > these 2 effects cancel, then my results should also apply to machines with > SATA drives and average processors. Using 8 drives (1-1) gets good > performance for teragen and terasort. Going to 12 drives (1.5 per core) > increases terasort performance by 15%. That might not seem like much > compared to increasing the number of drives by 50%, but a better comparison > is that 4 extra drives increased the cost of each machine by only about > 12%, so the extra drives are (barely) worth it. If you're more time > sensitive than cost sensitive, they they're definitely worth it. The extra > drives did not help teragen, apparently because both CPU and the internal > storage controller were close to saturation. So, of course everything > depends on the app. You're shooting for saturated CPUs and disk bandwidth. > Check that the CPU is not saturated (after checking Hadoop tuning and > optimizing the number of tasks). Check that you have enough memory for more > tasks with room leftover for a large buffer cache. Use 10 GbE networking > or make sure the network has enough headroom. Check the storage controller > can handle more bandwidth. If all are true (that is, no other > bottlenecks), consider adding more drives. > > > > Jeff > > > >> -----Original Message----- > >> From: Hank Cohen [mailto:hank.cohen@altior.com] > >> Sent: Friday, October 12, 2012 1:46 PM > >> To: user@hadoop.apache.org > >> Subject: RE: Spindle per Cores > >> > >> What empirical evidence is there for this rule of thumb? > >> In other words, what tests or metrics would indicate an optimal > >> spindle/core ratio and how dependent is this on the nature of the data > >> and of the map/reduce computation? > >> > >> My understanding is that there are lots of clusters with more spindles > >> than cores. Specifically, typical 2U servers can hold 12 3.5" disk > >> drives. So lots of Hadoop clusters have dual 4 core processors and 12 > >> spindles. Would it be better to have 6 core processors if you are > >> loading up the boxes with 12 disks? And most importantly, how would > >> one know that the mix was optimal? > >> > >> Hank Cohen > >> Altior Inc. > >> > >> -----Original Message----- > >> From: Patai Sangbutsarakum [mailto:silvianhadoop@gmail.com] > >> Sent: Friday, October 12, 2012 10:46 AM > >> To: user@hadoop.apache.org > >> Subject: Spindle per Cores > >> > >> I have read around about the hardware recommendation for hadoop > >> cluster. > >> One of them is recommend 1:1 ratio between spindle per core. > >> > >> Intel CPU come with Hyperthread which will double the number cores on > >> one physical CPU. eg. 8 cores with Hyperthread it because 16 which is > >> where we start to calculate about number of task slots per node. > >> > >> Once it come to spindle, i strongly believe I should pick 8 cores and > >> picks 8 disks in order to get 1:1 ratio. > >> > >> Please suggest > >> Patai > >> > > > > > > --f46d0447a10fc0364704cbe7e559 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Does hypertheading affect this ratio?

On Oct 12, 2012 9:36 PM, "Michael Segel&quo= t; <michael_segel@hotmail.c= om> wrote:
First, the obvious caveat... YMMV

Having said that.

The key here is to take a look across the various jobs that you will run. S= ome may be more CPU intensive, others more I/O intensive.

If you monitor these jobs via Ganglia, when you have too few spindles you s= hould see the wait cpu rise on the machines in the cluster. =A0That is to s= ay that you are putting an extra load on the systems because you're wai= ting for the disks to catch up.

If you increase the ratio of disks to CPU, you should see that load drop as= you are not wasting CPU cycles.

Note that its not just the number of spindles, but also the bus and the con= troller cards that can also affect the throughput of disk I/O.

Now just IMHO, there was a discussion on some of the CPU recommendations. T= o a point, it doesn't matter that much. You want to maximize the bang f= or the buck you can get w your hardware purchase.

Use the ratio as a buying guide. Fewer than a ratio of 1 disk per core, and= you're wasting the cpu that you bought.

Going higher than a ratio of 1, like 1.5, and you may be buying too many sp= indles and not see a performance gain that offsets your cost.

Search for a happy medium and don't sweat the maximum performance that = you may get.

HTH

On Oct 12, 2012, at 4:19 PM, Jeffrey Buell <jbuell@vmware.com> wrote:

> I've done some experiments along these lines. =A0I'm using hig= h-performance 15K RPM SAS drives instead of the more usual SATA drives, whi= ch should reduce the number of drives I need. =A0I have dual 4-core process= ors at 3.6 GHz. =A0These are more powerful than the average 4-core processo= r, which should increase the number of drives I need. =A0Assuming these 2 e= ffects cancel, then my results should also apply to machines with SATA driv= es and average processors. =A0Using 8 drives (1-1) gets good performance fo= r teragen and terasort. =A0Going to 12 drives (1.5 per core) increases tera= sort performance by 15%. =A0That might not seem like much compared to incre= asing the number of drives by 50%, but a better comparison is that 4 extra = drives increased the cost of each machine by only about 12%, so the extra d= rives are (barely) worth it. If you're more time sensitive than cost se= nsitive, they they're definitely worth it. =A0The extra drives did not = help teragen, apparently because both CPU and the internal storage controll= er were close to saturation. So, of course everything depends on the app. = =A0You're shooting for saturated CPUs and disk bandwidth. =A0Check that= the CPU is not saturated (after checking Hadoop tuning and optimizing the = number of tasks). Check that you have enough memory for more tasks with roo= m leftover for a large buffer cache. =A0Use 10 GbE networking or make sure = the network has enough headroom. =A0Check the storage controller can handle= more bandwidth. =A0If all are true (that is, no other bottlenecks), consid= er adding more drives.
>
> Jeff
>
>> -----Original Message-----
>> From: Hank Cohen [mailto:= hank.cohen@altior.com]
>> Sent: Friday, October 12, 2012 1:46 PM
>> To: user@hadoop.apache.o= rg
>> Subject: RE: Spindle per Cores
>>
>> What empirical evidence is there for this rule of thumb?
>> In other words, what tests or metrics would indicate an optimal >> spindle/core ratio and how dependent is this on the nature of the = data
>> and of the map/reduce computation?
>>
>> My understanding is that there are lots of clusters with more spin= dles
>> than cores. =A0Specifically, typical 2U servers can hold 12 3.5&qu= ot; disk
>> drives. =A0So lots of Hadoop clusters have dual 4 core processors = and 12
>> spindles. =A0Would it be better to have 6 core processors if you a= re
>> loading up the boxes with 12 disks? =A0And most importantly, how w= ould
>> one know that the mix was optimal?
>>
>> Hank Cohen
>> Altior Inc.
>>
>> -----Original Message-----
>> From: Patai Sangbutsarakum [mailto:silvianhadoop@gmail.com]
>> Sent: Friday, October 12, 2012 10:46 AM
>> To: user@hadoop.apache.o= rg
>> Subject: Spindle per Cores
>>
>> I have read around about the hardware recommendation for hadoop >> cluster.
>> One of them is recommend 1:1 ratio between spindle per core.
>>
>> Intel CPU come with Hyperthread which will double the number cores= on
>> one physical CPU. eg. 8 cores with Hyperthread it because 16 which= is
>> where we start to calculate about number of task slots per node. >>
>> Once it come to spindle, i strongly believe I should pick 8 cores = and
>> picks 8 disks in order to get 1:1 ratio.
>>
>> Please suggest
>> Patai
>>
>
>

--f46d0447a10fc0364704cbe7e559--