hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrzej Bialecki (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-490) Add ability to send "signals" to jobs and tasks
Date Tue, 29 Aug 2006 20:08:22 GMT
Add ability to send "signals" to jobs and tasks

                 Key: HADOOP-490
                 URL: http://issues.apache.org/jira/browse/HADOOP-490
             Project: Hadoop
          Issue Type: New Feature
          Components: mapred
    Affects Versions: 0.6.0
            Reporter: Andrzej Bialecki 

In some cases it would be useful to be able to "signal" a job and its tasks about some external
condition, or to broadcast a specific message to all tasks in a job. Currently we can only
send  a single pseudo-signal, that is to kill a job.

Example 1: some jobs may be gracefully terminated even if they didn't complete all their work,
e.g. Fetcher in Nutch may be running for a very long time if it blocks on relatively few sites
left over from the fetchlist. In such case it would be very useful to send it a message requesting
that it discards the rest of its input and gracefully completes its map tasks.

Example 2: available bandwidth for fetching may be different at different times of day, e.g.
daytime vs. nighttime, or total external link usage by other applications. Fetcher jobs often
run for several hours. It would be good to be able to send a "signal" to the Fetcher to throttle
or un-throttle its bandwidth usage depending on external conditions.

Job implementations could react to these messages either by implementing a method, or by registering
a listener, whichever seems more natural.

I'm not quite sure how to go about implementing it, I guess this would have to be a part of
 TaskUmbilicalProtocol but my knowledge here is a bit fuzzy ... ;) Comments are welcome.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message