falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1231) Improve JobCompletionNotification Service
Date Fri, 21 Aug 2015 10:53:45 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706541#comment-14706541

Pallavi Rao commented on FALCON-1231:

[~peeyushb], this is how the flow goes :
1. Oozie sends a JMS notification on Job status change. The notification is light with just
details on workflow Id, app name, status, etc. See details here -> http://oozie.apache.org/docs/4.0.0/DG_JMSNotifications.html
2. JMSMessageConsumer receives the message and creates a light WorkflowExecutionContext object
(with bare minimum details).
3. The context object is sent to WorkflowJobEndNotificationService. This service enhances
the context object with the rest of the information. This, it does by making a call to Oozie
and getting the jobInfo corresponding to the workflow Id. The job configuration (in jobInfo)
contains all the information required to populate the WorkflowExecutionContext.

The JMS message generated by FalconPostProcessing is also pretty much based off of the configuration
(actually, arguments whose values are passed in the configuration).

Hope this clarifies.

> Improve JobCompletionNotification Service
> -----------------------------------------
>                 Key: FALCON-1231
>                 URL: https://issues.apache.org/jira/browse/FALCON-1231
>             Project: Falcon
>          Issue Type: Sub-task
>            Reporter: Pallavi Rao
>            Assignee: Pallavi Rao
>         Attachments: FALCON-1231.patch, FALCON-1231.patch, OozieJobStatusNotification.pdf
> Workflow completion is currently dependent on post processing action putting messages
(success/failure) to a JMS queue.
> When failures occur on Hadoop / Oozie, the post processing action may not  get executed
either. In such cases, Falcon will be blind to the status of the workflow and re-tries will
not happen.
> Explore url notification of oozie to get job completion notifications. Fall back on polling
when workflows run beyond SLA.

This message was sent by Atlassian JIRA

View raw message