Return-Path: Delivered-To: apmail-lucene-hadoop-user-archive@locus.apache.org Received: (qmail 68454 invoked from network); 7 Nov 2007 11:39:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 7 Nov 2007 11:39:24 -0000 Received: (qmail 9355 invoked by uid 500); 7 Nov 2007 11:39:12 -0000 Delivered-To: apmail-lucene-hadoop-user-archive@lucene.apache.org Received: (qmail 8752 invoked by uid 500); 7 Nov 2007 11:39:11 -0000 Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-user@lucene.apache.org Delivered-To: mailing list hadoop-user@lucene.apache.org Received: (qmail 8743 invoked by uid 99); 7 Nov 2007 11:39:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Nov 2007 03:39:10 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dgottfrid@gmail.com designates 64.233.162.236 as permitted sender) Received: from [64.233.162.236] (HELO nz-out-0506.google.com) (64.233.162.236) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Nov 2007 11:39:14 +0000 Received: by nz-out-0506.google.com with SMTP id i28so1519439nzi for ; Wed, 07 Nov 2007 03:38:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=pfvc/PIlD5C+OVqt5/AOduAlVPVAnQXBzjgI+wnGkEc=; b=GorGtded5M2tU1boGsHo1HDVr4UHERqIhmCfYzA5zgvMnHoQTBpBzIN8K7PesY6c25XbfJQerSseZP//txzfawYUJxwX1E2XIB1xvDI4OhSOdSUBJDbZw3LEvM3tyHbihVtKeDtM8k6r8r2FvnU6TTnaBdbLWQpuX+KHpL8YeW8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=UVDqZzzwMfj8tYdwx0PPvpuydGGcDiFk7OypXgtEqd0UQ7jdkVmsOay2lXQ0pwlvOLe5IpNXwtvxwkkEGMIs0bCs4xjdAHNNdzmbRjxGSqTFuxEcJAIRIyNM2wtVjXt/gH9uQALQH537zWIv21FhDJsqfK0q0hgnCcncY7KczXM= Received: by 10.142.80.7 with SMTP id d7mr1959580wfb.1194435532543; Wed, 07 Nov 2007 03:38:52 -0800 (PST) Received: by 10.142.246.17 with HTTP; Wed, 7 Nov 2007 03:38:52 -0800 (PST) Message-ID: <938caae40711070338h81cc347x32f497cb85a246d3@mail.gmail.com> Date: Wed, 7 Nov 2007 06:38:52 -0500 From: "Derek Gottfrid" Reply-To: derek@codecubed.com Sender: dgottfrid@gmail.com To: hadoop-user@lucene.apache.org Subject: Re: configuration suggestions for 1k nodes In-Reply-To: <47315946.7050301@yahoo-inc.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <938caae40711062003i5ac37b53u8054c879403e5330@mail.gmail.com> <47315946.7050301@yahoo-inc.com> X-Google-Sender-Auth: 5c63c4fe4dd2c374 X-Virus-Checked: Checked by ClamAV on apache.org excellent. I will keep the group posted as I progress. On 11/7/07, Raghu Angadi wrote: > > Not a complete list by far, but just a start : > > For HDFS: > > - Make sure you run Java 6 (jdk1.6). > - Set namenode handler count to 40 or more (dfs.namenode.handler.count, > and may be mapred.job.tracker.handler.count etc). > > - more config guides are in the works : > https://issues.apache.org/jira/browse/HADOOP-1917 > > Raghu. > > Derek Gottfrid wrote: > > Are there configuration suggestions for 1k nodes ? I was seeing tons > > of timeouts trying to run 1k nodes. Are there network settings that I > > need to make? Out of the box stuff seemed to work up to a couple > > hundred but I want to go bigger. Pointers/Suggestions? > > > > derek > > > > > > ps: i wrote up my ec2/hadoop at nytimes.com - check it out > > http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super-computing-fun/ > >