hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yoram Arnon (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-491) there should be a protocol for a map or reduce task to tell the jobtracker its alive
Date Thu, 31 Aug 2006 07:52:24 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-491?page=all ]

Yoram Arnon updated HADOOP-491:
-------------------------------

    Component/s: contrib/streaming
                     (was: mapred)
    Description: 
The jobtracker relies on task to send heartbeats  to know the tasks are still alive.
There is a 600 seconds timeout preset.
hadoop streaming also uses output from the program it spawns to indicate progress, sending
appropriate heartbeats.
Some spawned programs spend longer that 600 seconds without any output while being perfectly
healthy.

It would be good to enhance the interface between hadoop streaming and the programs it spawns
to track a healthy program in the absense of output.

There are certain dangers with this protocol: e.g. a task can run a separate thread that does
nothing but send "i'm alive" message.   This would be a user bug to abuse the API in such
way.  

  was:
Currently, the jobtracker relies on task to write something to the output to know the tasks
are still alive.
There is a 600 seconds timeout preset.
Some tasks spend longer that 600 seconds without any output while being perfectly healthy.

It would be good to have an API for a task to explicitly tell the jobtracker it is alive.

There are certain dangers with this protocol: e.g. a task can run a separate thread that does
nothing but send "i'm alive" message.   This would be a user bug to abuse the API in such
way.  


> there should be a protocol for a map or reduce task to tell the jobtracker its alive
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-491
>                 URL: http://issues.apache.org/jira/browse/HADOOP-491
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: contrib/streaming
>            Reporter: arkady borkovsky
>
> The jobtracker relies on task to send heartbeats  to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses output from the program it spawns to indicate progress, sending
appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly
healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it
spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that
does nothing but send "i'm alive" message.   This would be a user bug to abuse the API in
such way.  

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message