Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: neutral (asf.osuosl.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=mime-version:in-reply-to:references:content-type:message-id:
	content-transfer-encoding:from:subject:date:to:x-mailer;
	b=eLsLBIkmHvi+rPtrMdBe1fONpZ8Ln4JbDKW6XJrkt9Od3ChFrK2gOTX4glTuDY+V
Mime-Version: 1.0 (Apple Message framework v750)
In-Reply-To: <4475D3E4.3020507@apache.org>
References: <5438AA87-1469-4F49-BABF-43E3A6BD1856@gmail.com>
 <D878EB96-A840-4CE0-B8B9-156778BBD6C3@mac.com>
 <44746E29.1030803@dragonflymc.com>
 <563DFE2F-2EEE-4DB8-AE68-268806A7018E@gmail.com>
 <4475D3E4.3020507@apache.org>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <45BAF9F0-D7FE-428C-B801-3FA554B4D4F1@yahoo-inc.com>
Content-Transfer-Encoding: 7bit
From: Ben Reed <breed@yahoo-inc.com>
Subject: Re: Multiple tasktrackers per node
Date: Thu, 25 May 2006 09:19:37 -0700
To: hadoop-user@lucene.apache.org

My task_zoom.patch fixes "the 10 sec delay before getting another  
task when a task completes" bug. It is a rather minor part of the  
task_zoom.patch. Basically, the TaskTracker updates the JobTracker as  
soon as the task completes. There was another bug in the JobTracker  
that made it count all tasks rather than just the running tasks,  
which could cause a delay longer than 10 secs in some cases that the  
patch fixes.

ben

On May 25, 2006, at 8:57 AM, Doug Cutting wrote:

> Gianlorenzo Thione wrote:
>> Thanks for the answer. So far I am still trying to understand how   
>> each tasktracker gets multiple map or reduce tasks to be executed   
>> simultaneously. I have run a simple job with 53 map tasks on 5  
>> nodes,  and at all times each node was executing a single task.  
>> Each cluster  node is a 4 core machine, so theoretically this was  
>> a 16-node cluster  and I feel that the resources were actually  
>> underutilized. Am I  missing something? Is there a parameter for a  
>> minimum number of tasks  to be executed in parallel (I found a  
>> parameter for setting a maximum  [which I set to 4])? If I run 4  
>> TaskTrackers per node then each node  gets a map task at the same  
>> time and execution seems overall much  faster.
>
> The task tracker can currently get starved for work when tasks  
> complete too quickly.  This is a bug that will hopefully be fixed  
> soon.  The problem is that the task tracker only polls for a new  
> task once per heartbeat (10 seconds).  Instead it should poll for  
> new tasks as soon as tasks complete.  As a short-term workaround  
> you can decrease the heartbeat interval to one second in  
> MRConstants.java.  With smaller clusters (< 100 machines) that  
> should not cause any problems.
>
> Doug
>
>