hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michel Tourn (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-556) Streaming should send keep-alive signals to Reporter every 10 seconds, not every 100 records
Date Sat, 23 Sep 2006 00:27:22 GMT
Streaming should send keep-alive signals to Reporter every 10 seconds, not every 100 records
--------------------------------------------------------------------------------------------

                 Key: HADOOP-556
                 URL: http://issues.apache.org/jira/browse/HADOOP-556
             Project: Hadoop
          Issue Type: Improvement
          Components: contrib/streaming
            Reporter: Michel Tourn


Streaming should send keep-alive signals to Reporter every 10 seconds, not every 100 records.
A first version of this was already implemented but was not satisfactory.

Now we check whether 10sec have passed:

-after reading a *line* of Application's stderr 
 (so please use as your stderr heartbeat: cerr << ".\n", not cerr << "."  )

-and after outputting a record in mapper / combiner / reducer.

If 10 sec have passed then we set Reporter status.

Effects:

-the reporter status changes more often and provides useful feedback in the Web UI or in another
client.
-a Task will not time out after 10 minutes just because it outputs records slowly. 

No artificial heartbeat is introduced in this proposal.
The streaming Application still has to show activity (either on stdout or on stderr)









-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message