hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-580) Job setup and take down on Nodes
Date Fri, 06 Oct 2006 16:14:20 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-580?page=comments#action_12440501 ] 
Doug Cutting commented on HADOOP-580:

It sounds like you want to run user code in the TaskTracker.  Right now, user code only runs
in per-task child processes.  We also run some in the JobTracker, but would like to get rid
of that, so that no long-running daemons run user code, as discussed in the following thread:


> Job setup and take down on Nodes
> --------------------------------
>                 Key: HADOOP-580
>                 URL: http://issues.apache.org/jira/browse/HADOOP-580
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Benjamin Reed
> It would be nice if there was a hook for doing job provisioning and cleanup on compute
nodes. The TaskTracker implicitly knows when a job starts (a task for the job is received)
and pollForTaskWithClosedJob() will explicitly say that a job is finished if a Map task has
been run (If only Reduce tasks have run and are finished I don't think pollForTaskWithClosedJob()
will return anything will it?), but child Tasks do not get this information.
> It would be nice if there was a hook so that programmers could do some provisioning when
a job starts and cleanup when a job ends. Caching addresses some of the provisioning, but
in some cases a helper daemon may need to be started or the results of queries need to be
retrieved and having startJob(), finishJob() callbacks that happen exactly once for each node
that runs part of the job would be wonderful.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message