hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23
Date Mon, 19 Sep 2011 16:05:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107931#comment-13107931
] 

Robert Joseph Evans commented on MAPREDUCE-3028:
------------------------------------------------

@Alejandro, OK I agree completely that an RSS like feed for reporting which jobs have completed
is preferable, so long as we can make it so there is relatively little load on the system
even when polling multiple times a second.

But if we remove the notification all together there may be other applications relying on
this functionality that fail.  I would prefer to see both go in, and we deprecate the push.
 This would give users time to move off of it.

Also the delay is not that bad if it is the application master that is sending the notification.
 The issue with the delay before was primarily caused by the notification being single threaded.
 If one application failed with a 60sec time out then the other notifications started to pile
up behind it.  This would still be bad for a map/reduce job to pause 60 seconds for the notification
to go through, especially with retry, but it would not impact any other applications not connected
to it.

> Support job end notification in .next /0.23
> -------------------------------------------
>
>                 Key: MAPREDUCE-3028
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Mohammad Kamrul Islam
>             Fix For: 0.23.0
>
>
> Oozie primarily depends on  the job end notification to determine when the job finishes.
In the current version,  job end notification is implemented in job tracker. Since job tracker
will be removed in the upcoming hadoop release (.next), we wander where this support will
move. I think this best effort notification could be implemented in the new Application Manager
as one of the last step of job completion.
> Whatever implementation will it be, Oozie badly needs this feature to be continued in
next releases as well.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message