Return-Path: Delivered-To: apmail-hadoop-core-commits-archive@www.apache.org Received: (qmail 23876 invoked from network); 20 May 2009 13:59:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 20 May 2009 13:59:56 -0000 Received: (qmail 81551 invoked by uid 500); 20 May 2009 14:00:08 -0000 Delivered-To: apmail-hadoop-core-commits-archive@hadoop.apache.org Received: (qmail 81505 invoked by uid 500); 20 May 2009 14:00:08 -0000 Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-commits@hadoop.apache.org Received: (qmail 81467 invoked by uid 99); 20 May 2009 14:00:08 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 May 2009 14:00:08 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.130] (HELO eos.apache.org) (140.211.11.130) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 May 2009 13:59:57 +0000 Received: from eos.apache.org (localhost [127.0.0.1]) by eos.apache.org (Postfix) with ESMTP id 21788118BF for ; Wed, 20 May 2009 13:59:36 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki To: core-commits@hadoop.apache.org Date: Wed, 20 May 2009 13:59:35 -0000 Message-ID: <20090520135935.12524.79188@eos.apache.org> Subject: [Hadoop Wiki] Update of "LargeClusterTips" by SteveLoughran X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification. The following page has been changed by SteveLoughran: http://wiki.apache.org/hadoop/LargeClusterTips The comment on the change is: More big cluster tips ------------------------------------------------------------------------------ Below are tips for managing large clusters. + * Have a good sysadmin if you're not one yourself. * Take a look at a presentation done by Allen Wittenauer from Yahoo!: http://tinyurl.com/5foamm + * Have the LAN closed off to untrusted users. This simplifies security. + * Use LDAP or similar to manage user accounts. - * Only put the slaves file on your namenode and secondary namenode to prevent confusion + * Only put the slaves file on your namenode and secondary namenode to prevent confusion. + * Have identical hardware on all machines in the cluster, eliminating the need to have different + configuration options (task slots, data directory locations, etc) + * Use RPMs to install the Hadoop binaries. Self:Cloudera provide some RPMs for this, and a web site to generate configuration RPM files. + * Use kickstart or similar to bring up the machines. - * Use a system configuration management package to keep Hadoop's source consistent across all nodes. Some example packages are bcfg2, smartfrog, puppet, cfengine, etc. + * Consider a system configuration management package to keep Hadoop's source and configuration consistent across all nodes. Some example packages are bcfg2, smartfrog, puppet, cfengine, etc. - * Have a good sysadmin if you're not one + * If you are trying to configure the machines one by one, step away from the keyboard. That is not the way to manage a cluster. See the Self:AmazonEC2 and AmazonS3 pages for tips on managing clusters built on EC2 and S3. - Other good documentation: http://wiki.smartfrog.org/wiki/display/sf/Patterns+of+Hadoop+Deployment + Other good documentation: [http://wiki.smartfrog.org/wiki/display/sf/Patterns+of+Hadoop+Deployment Patterns of Hadoop Deployment]