hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23
Date Mon, 19 Sep 2011 15:49:09 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107921#comment-13107921
] 

Alejandro Abdelnur commented on MAPREDUCE-3028:
-----------------------------------------------

@Robert, with MR1 we are doing push. A lesson learned from that is that if the receivers of
the call are slow or not avail the push notifications pile up and there is a long delay before
they are delivered (at 60 secs HTTP timeout per bad notification, this is a lot in a heavy
loaded system). Using a RSS (like) feed the amount of memory used in the JT is about the same
(I would think even less) and there is not contention & delay. Plus the fact that more
than one party can 'poll' for the status of a job as opposed to a single party 'push'. Using
something like size-bound concurrent list to store & serve the job status would not create
contention in the JT. Clients could poll often (30 to 60 secs) which seems like a minimal
delay.

> Support job end notification in .next /0.23
> -------------------------------------------
>
>                 Key: MAPREDUCE-3028
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Mohammad Kamrul Islam
>             Fix For: 0.23.0
>
>
> Oozie primarily depends on  the job end notification to determine when the job finishes.
In the current version,  job end notification is implemented in job tracker. Since job tracker
will be removed in the upcoming hadoop release (.next), we wander where this support will
move. I think this best effort notification could be implemented in the new Application Manager
as one of the last step of job completion.
> Whatever implementation will it be, Oozie badly needs this feature to be continued in
next releases as well.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message