Return-Path: 
 <hadoop-user-return-3097-apmail-lucene-hadoop-user-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-hadoop-user-archive@locus.apache.org
Received: (qmail 51658 invoked from network); 16 Nov 2007 18:46:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 16 Nov 2007 18:46:14 -0000
Received: (qmail 99971 invoked by uid 500); 16 Nov 2007 18:45:59 -0000
Delivered-To: apmail-lucene-hadoop-user-archive@lucene.apache.org
Received: (qmail 99941 invoked by uid 500); 16 Nov 2007 18:45:59 -0000
Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hadoop-user-help@lucene.apache.org>
List-Unsubscribe: <mailto:hadoop-user-unsubscribe@lucene.apache.org>
List-Post: <mailto:hadoop-user@lucene.apache.org>
List-Id: <hadoop-user.lucene.apache.org>
Reply-To: hadoop-user@lucene.apache.org
Delivered-To: mailing list hadoop-user@lucene.apache.org
Received: (qmail 99932 invoked by uid 99); 16 Nov 2007 18:45:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Nov 2007 10:45:59 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Nov 2007 18:45:47 +0000
Received: from arunc.eglbp.corp.yahoo.com (arunc.eglbp.corp.yahoo.com
 [10.66.74.226])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id lAGIj0VC073485;
	Fri, 16 Nov 2007 10:45:01 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-authentication-warning:date:from:to:subject:
	message-id:references:mime-version:content-type:
	content-disposition:in-reply-to:user-agent;
	b=gSnpIAB5bPAVMVcFgbu2Ara0neI5Uy7n7qa4JSmSGw1hJggQ7SMePpFvcF+kU4at
Received: (from arunc@localhost)
	by arunc.eglbp.corp.yahoo.com (8.13.3/8.13.3) id lAGIixbq025302;
	Sat, 17 Nov 2007 00:14:59 +0530 (IST)
	(envelope-from arunc@yahoo-inc.com)
X-Authentication-Warning: arunc.eglbp.corp.yahoo.com: arunc set sender to
 arunc@yahoo-inc.com using -f
Date: Sat, 17 Nov 2007 00:14:58 +0530
From: Arun C Murthy <arunc@yahoo-inc.com>
To: hadoop-user@lucene.apache.org, jdoklovic@ibsys.com
Subject: Re: performance test tips?
Message-ID: <20071116184458.GA25178@yahoo-inc.com>
References: <1195236021.4829.7.camel@jd-linux.ibsys.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1195236021.4829.7.camel@jd-linux.ibsys.com>
User-Agent: Mutt/1.4.2.1i
X-Virus-Checked: Checked by ClamAV on apache.org

Jonathan

On Fri, Nov 16, 2007 at 12:00:21PM -0600, jonathan doklovic wrote:
>Hi,
>
>We've finally got our hadoop cluster up, some data to crunch and a
>map/reduce job.
>
>After running a few configurations, i'm not sure about our performance
>and would like to get some advice....
>
>We have a 20 node ec2 cluster.
>We have 750MB of data.
>currently our job seems to be doing 1%/min on the cluster.
>Using a much smaller subset of data and running locally, the job takes a
>matter of seconds.
>
>
>Here's our hadoop-site.xml
>
>
><property>
>  <name>mapred.tasktracker.tasks.maximum</name>
>  <value>20</value>
></property>
>

That is very high, you are basically letting the TT spawn 40 child-jvms. I wouldn't go above 3 or 4 for that config.

Hopefully http://lucene.apache.org/hadoop/cluster_setup.html is useful...

Arun