hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23
Date Tue, 18 Oct 2011 17:13:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129864#comment-13129864
] 

Robert Joseph Evans commented on MAPREDUCE-3028:
------------------------------------------------

I don't like the job history server doing it because launching the job history server is some
what optional (The default start scripts do not launch it currently).  Also it will add in
a new delay, and a lot more churn on the Job history server.  Currently the JobHistory server
polls HDFS for new jobs that have finished.  If the job has finished it will move it to its
own directory, and then cache some info about the app based only on the name of the jhist
file.  If we want this to do notification also then we need to either store the URL in the
jhist file name, or force the server to read in the config file every time it pulls in a new
Application.

I agree that having applications poll for status with something like a very low latency RSS
is the preferable solution and as such I would like to see the minimal done to support backwards
compatibility, which would be to put the notification in the ApplicationMaster.  But it is
Ravi that is doing the fix and if he wants to do more then that and it is not something that
is going to cause more problems then it fixes then I would say let him do it.  

What I mean by cause more problems then it fixes is that we have had lots of issues in production
with having the notification be in a single thread blocking other notifications.  What ever
solution it is it cannot allow other notifications to become blocked or severely delayed by
a single bad URL. (Or our case of a remote server crashing in the middle of the request and
some how causing the socket be in a psudo state where the read blocks indefinitely and there
is no one on the other side that is ever going to write)


                
> Support job end notification in .next /0.23
> -------------------------------------------
>
>                 Key: MAPREDUCE-3028
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Mohammad Kamrul Islam
>            Assignee: Ravi Prakash
>            Priority: Blocker
>             Fix For: 0.23.0
>
>
> Oozie primarily depends on  the job end notification to determine when the job finishes.
In the current version,  job end notification is implemented in job tracker. Since job tracker
will be removed in the upcoming hadoop release (.next), we wander where this support will
move. I think this best effort notification could be implemented in the new Application Manager
as one of the last step of job completion.
> Whatever implementation will it be, Oozie badly needs this feature to be continued in
next releases as well.
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message