hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkady Borkovsky <ark...@yahoo-inc.com>
Subject Re: [jira] Commented: (HADOOP-580) Job setup and take down on Nodes
Date Sat, 07 Oct 2006 23:42:45 GMT
It would be good to have this process to be an arbitrary executable,  
not necessarily a Java program.

On Oct 6, 2006, at 10:49 AM, Benjamin Reed (JIRA) wrote:

>     [  
> http://issues.apache.org/jira/browse/HADOOP-580? 
> page=comments#action_12440557 ]
>
> Benjamin Reed commented on HADOOP-580:
> --------------------------------------
>
> No. I'm very against running code in the Trackers (as my mail  
> indicates :). The idea would be that you would spawn off a child  
> process at the beginning of a job and kill it at the end. (Or some  
> variation on that theme.)
>
>> Job setup and take down on Nodes
>> --------------------------------
>>
>>                 Key: HADOOP-580
>>                 URL: http://issues.apache.org/jira/browse/HADOOP-580
>>             Project: Hadoop
>>          Issue Type: New Feature
>>          Components: mapred
>>            Reporter: Benjamin Reed
>>
>> It would be nice if there was a hook for doing job provisioning and  
>> cleanup on compute nodes. The TaskTracker implicitly knows when a job  
>> starts (a task for the job is received) and  
>> pollForTaskWithClosedJob() will explicitly say that a job is finished  
>> if a Map task has been run (If only Reduce tasks have run and are  
>> finished I don't think pollForTaskWithClosedJob() will return  
>> anything will it?), but child Tasks do not get this information.
>> It would be nice if there was a hook so that programmers could do  
>> some provisioning when a job starts and cleanup when a job ends.  
>> Caching addresses some of the provisioning, but in some cases a  
>> helper daemon may need to be started or the results of queries need  
>> to be retrieved and having startJob(), finishJob() callbacks that  
>> happen exactly once for each node that runs part of the job would be  
>> wonderful.
>
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the  
> administrators:  
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:  
> http://www.atlassian.com/software/jira
>
>


Mime
View raw message