hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrzej Bialecki (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-490) Add ability to send "signals" to jobs and tasks
Date Tue, 19 Sep 2006 07:47:25 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-490?page=comments#action_12435717 ] 
Andrzej Bialecki  commented on HADOOP-490:

Re: namenode issue: yes, that's a good point - I didn't think of that, mainly because I'm
working with smaller clusters (dozens machines at most).

Re: cost of scanning: that's true as well, although tasks don't have to poll so often, in
some cases you could configure the poll interval to be in the range of minutes. However, this
points back to a deficiency in the current framework, namely that there is no support for
sending arbitrary messages to tasks. If there were a way to do this (well, then the issue
would be solved and we wouldn't need this MQ api ... ;) then we could run a separate filesystem
monitor, which would dispatch events to all listening tasks concerning filesystem changes
such as file/dir creation/deletion/update.

Overall, I'm aware that this is a less than ideal solution to the problem - IMHO my original
proposal explained in this issue would be better. ;)

> Add ability to send "signals" to jobs and tasks
> -----------------------------------------------
>                 Key: HADOOP-490
>                 URL: http://issues.apache.org/jira/browse/HADOOP-490
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.6.0
>            Reporter: Andrzej Bialecki 
> In some cases it would be useful to be able to "signal" a job and its tasks about some
external condition, or to broadcast a specific message to all tasks in a job. Currently we
can only send  a single pseudo-signal, that is to kill a job.
> Example 1: some jobs may be gracefully terminated even if they didn't complete all their
work, e.g. Fetcher in Nutch may be running for a very long time if it blocks on relatively
few sites left over from the fetchlist. In such case it would be very useful to send it a
message requesting that it discards the rest of its input and gracefully completes its map
> Example 2: available bandwidth for fetching may be different at different times of day,
e.g. daytime vs. nighttime, or total external link usage by other applications. Fetcher jobs
often run for several hours. It would be good to be able to send a "signal" to the Fetcher
to throttle or un-throttle its bandwidth usage depending on external conditions.
> Job implementations could react to these messages either by implementing a method, or
by registering a listener, whichever seems more natural.
> I'm not quite sure how to go about implementing it, I guess this would have to be a part
of  TaskUmbilicalProtocol but my knowledge here is a bit fuzzy ... ;) Comments are welcome.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message