Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
DomainKey-Status: good
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=mime-version:in-reply-to:references:content-type:message-id:
	content-transfer-encoding:from:subject:date:to:x-mailer;
	b=2bN5xEoSj1SFYn1UpQGLLIx3Q2zDD/7paaI7wzZSGSK+MVq4omUnBWvEtdOU24wd
Mime-Version: 1.0 (Apple Message framework v624)
In-Reply-To: <30184018.1160156962192.JavaMail.root@brutus>
References: <30184018.1160156962192.JavaMail.root@brutus>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <406812e83daf6ae6d3387c4ea1df3818@yahoo-inc.com>
Content-Transfer-Encoding: 7bit
From: Arkady Borkovsky <arkady@yahoo-inc.com>
Subject: Re: [jira] Commented: (HADOOP-580) Job setup and take down on Nodes
Date: Sat, 7 Oct 2006 16:42:45 -0700
To: hadoop-dev@lucene.apache.org

It would be good to have this process to be an arbitrary executable,  
not necessarily a Java program.

On Oct 6, 2006, at 10:49 AM, Benjamin Reed (JIRA) wrote:

>     [  
> http://issues.apache.org/jira/browse/HADOOP-580? 
> page=comments#action_12440557 ]
>
> Benjamin Reed commented on HADOOP-580:
> --------------------------------------
>
> No. I'm very against running code in the Trackers (as my mail  
> indicates :). The idea would be that you would spawn off a child  
> process at the beginning of a job and kill it at the end. (Or some  
> variation on that theme.)
>
>> Job setup and take down on Nodes
>> --------------------------------
>>
>>                 Key: HADOOP-580
>>                 URL: http://issues.apache.org/jira/browse/HADOOP-580
>>             Project: Hadoop
>>          Issue Type: New Feature
>>          Components: mapred
>>            Reporter: Benjamin Reed
>>
>> It would be nice if there was a hook for doing job provisioning and  
>> cleanup on compute nodes. The TaskTracker implicitly knows when a job  
>> starts (a task for the job is received) and  
>> pollForTaskWithClosedJob() will explicitly say that a job is finished  
>> if a Map task has been run (If only Reduce tasks have run and are  
>> finished I don't think pollForTaskWithClosedJob() will return  
>> anything will it?), but child Tasks do not get this information.
>> It would be nice if there was a hook so that programmers could do  
>> some provisioning when a job starts and cleanup when a job ends.  
>> Caching addresses some of the provisioning, but in some cases a  
>> helper daemon may need to be started or the results of queries need  
>> to be retrieved and having startJob(), finishJob() callbacks that  
>> happen exactly once for each node that runs part of the job would be  
>> wonderful.
>
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the  
> administrators:  
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:  
> http://www.atlassian.com/software/jira
>
>