hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Reed (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-580) Job setup and take down on Nodes
Date Fri, 06 Oct 2006 17:49:22 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-580?page=comments#action_12440557 ] 
Benjamin Reed commented on HADOOP-580:

No. I'm very against running code in the Trackers (as my mail indicates :). The idea would
be that you would spawn off a child process at the beginning of a job and kill it at the end.
(Or some variation on that theme.)

> Job setup and take down on Nodes
> --------------------------------
>                 Key: HADOOP-580
>                 URL: http://issues.apache.org/jira/browse/HADOOP-580
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Benjamin Reed
> It would be nice if there was a hook for doing job provisioning and cleanup on compute
nodes. The TaskTracker implicitly knows when a job starts (a task for the job is received)
and pollForTaskWithClosedJob() will explicitly say that a job is finished if a Map task has
been run (If only Reduce tasks have run and are finished I don't think pollForTaskWithClosedJob()
will return anything will it?), but child Tasks do not get this information.
> It would be nice if there was a hook so that programmers could do some provisioning when
a job starts and cleanup when a job ends. Caching addresses some of the provisioning, but
in some cases a helper daemon may need to be started or the results of queries need to be
retrieved and having startJob(), finishJob() callbacks that happen exactly once for each node
that runs part of the job would be wonderful.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message