hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-491) streaming jobs should allow programs that don't do any IO for a long time
Date Tue, 06 Feb 2007 05:59:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470453
] 

Owen O'Malley commented on HADOOP-491:
--------------------------------------

That did happen and it helped. Because streaming can use binaries that the user doesn't control,
they don't always have the option to add printing bytes to stderr. Furthermore, as we start
sending more of the output back to the user's console, it is less clear that having the application
send data to stderr is a good idea.

> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-491
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: contrib/streaming
>            Reporter: arkady borkovsky
>         Assigned To: Arun C Murthy
>             Fix For: 0.12.0
>
>         Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats  to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate
progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly
healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it
spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that
does nothing but send "i'm alive" message.   This would be a user bug to abuse the API in
such way.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message