Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 9485 invoked from network); 29 Sep 2009 15:16:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 29 Sep 2009 15:16:07 -0000 Received: (qmail 30891 invoked by uid 500); 29 Sep 2009 15:16:05 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 30816 invoked by uid 500); 29 Sep 2009 15:16:05 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 30806 invoked by uid 99); 29 Sep 2009 15:16:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 15:16:05 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 15:15:54 +0000 Received: from localhost (localhost [127.0.0.1]) by tobor.hpl.hp.com (Postfix) with ESMTP id B64B2B7E30 for ; Tue, 29 Sep 2009 16:15:33 +0100 (BST) X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com Received: from tobor.hpl.hp.com ([127.0.0.1]) by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 9mXwcPDlejCl for ; Tue, 29 Sep 2009 16:15:27 +0100 (BST) Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tobor.hpl.hp.com (Postfix) with ESMTPS id E3277B7E2F for ; Tue, 29 Sep 2009 16:15:25 +0100 (BST) MailScanner-NULL-Check: 1254842112.36194@Ccvi2HT3QXGv4xY1zxrAHw Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158]) by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id n8TFFBQI006791 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 29 Sep 2009 16:15:12 +0100 (BST) Message-ID: <4AC2247F.9090306@apache.org> Date: Tue, 29 Sep 2009 16:15:11 +0100 From: Steve Loughran User-Agent: Thunderbird 2.0.0.23 (X11/20090812) MIME-Version: 1.0 To: common-user@hadoop.apache.org Subject: Re: How 'commodity' is 'commodity' References: <1254195954.3354.28.camel@Cheyenne> <279a90ff0909290055x84cbdd1w743f1e4cd3f4d51a@mail.gmail.com> <4AC1DB12.5020807@apache.org> <4658c60909290309v4927f908u9c8c2bd3e28a0d71@mail.gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-HPL-MailScanner-Information: Please contact the ISP for more information X-MailScanner-ID: n8TFFBQI006791 X-HPL-MailScanner: Found to be clean X-HPL-MailScanner-From: stevel@apache.org X-Virus-Checked: Checked by ClamAV on apache.org Edward Capriolo wrote: > In hadoop terms commodity means "Not super computer". If you look > around most large deployments have DataNodes with duel quad core > processors 8+GB ram and numerous disks, that is hardly the PC you find > under your desk. I have 4 cores and 6GB RAM, but only one HDD on the desk. That ram gets sucked up by the memory hogs: IDE, firefox and VMWare. The fact that VMWare uses less RAM to host an entire OS image than firefox shows that you can get away with virtualised work, or that firefox is overweight. 18862 20 0 2139m 1.4g 48m S 0 25.0 46:57.81 java 12822 20 0 1050m 476m 18m S 3 8.1 53:34.39 firefox 13037 20 0 949m 386m 365m S 11 6.5 206:34.70 vmware-vmx 20932 20 0 688m 374m 11m S 0 6.4 2:05.18 java > For example, we had a dev clusters is a very modest setup, 5 HP DL > 145. Duel Core 4 GB RAM, 2 SATA DISKS. > > I did not do any elaborate testing, but i found that: > > !!!!One!!!! DL180G5 (2x Quad Code) (8GB ram) 8 SATA disk crushed the 5 > node cluster. So it is possible but it might be more administrative > work then it is worth. Interesting. Is this CPU or IO intensive work?