falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peeyush Bishnoi (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FALCON-1313) Generate notification for Falcon job status
Date Tue, 21 Jul 2015 15:45:04 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635286#comment-14635286
] 

Peeyush Bishnoi edited comment on FALCON-1313 at 7/21/15 3:44 PM:
------------------------------------------------------------------

[~pallavi.rao] Please see my answers inline:
{quote}
Approach 1 : Extend the workflow by leveraging Oozie email action.
1. If the users are expected to configure the SMTP server in Oozie, why not just let them
add an email notification in their user workflow itself? We'll end up taking in the same set
of parameters anyway.
{quote}

In most of the cases users are not allowed to configure Oozie services that has to be specified
in oozie-site.xml. AFAIK, it is the administrator who configure the Oozie services for users.
So, we are assuming that users are not expected to configure the SMTP service in Oozie. Also
we should ensure that users should not feel pain in adding XML tags for email notification
in their workflow. 

{quote}
Approach 2 : Create notification listener to capture the job status through JMSMessageConsumer
class.
1. Everything is fine when all sub-workflows and sub-actions succeed. But, when post-processing
fails (user workflow may have succeeded), there will be no JMS notification, hence, Falcon
will not be aware of the status of that workflow. In fact, Falcon suffers from the same problem
in case of retries too. At InMobi, we have seen that retries from Falcon side usually don't
happen because the JMS notification never came due to failure. This is what I meant when I
said, the JMS notifications are not reliable.
{quote}

If succeeded post processing fails due to some reason and if there is no JMS notification,
many listener services in Falcon (AbstractRerunHandler, CatalogPartitionHandler, LateRerunHandler,
MetadataMappingService, RetryHandler) will be impacted which depend on JMS notification and
this need to be fixed in Falcon. But it does not mean that we should not add other service.


was (Author: peeyushb):
[~pallavi.rao] Please see my answers inline:
{quote}
Approach 1 : Extend the workflow by leveraging Oozie email action.
1. If the users are expected to configure the SMTP server in Oozie, why not just let them
add an email notification in their user workflow itself? We'll end up taking in the same set
of parameters anyway.
{quote}

In most of the cases users are not allowed to configure Oozie services that has to be specified
in oozie-site.xml. AFAIK, it is the administrator who configure the Oozie services for users.
So, we are assuming that users are not expected to configure the SMTP service in Oozie. Also
we should ensure that users should not feel pain in adding XML tags for email notification
in their workflow. 

{quote}
Approach 2 : Create notification listener to capture the job status through JMSMessageConsumer
class.
1. Everything is fine when all sub-workflows and sub-actions succeed. But, when post-processing
fails (user workflow may have succeeded), there will be no JMS notification, hence, Falcon
will not be aware of the status of that workflow. In fact, Falcon suffers from the same problem
in case of retries too. At InMobi, we have seen that retries from Falcon side usually don't
happen because the JMS notification never came due to failure. This is what I meant when I
said, the JMS notifications are not reliable.
{quote}

If succeeded post processing fails due to some reason and if there is no JMS notification,
many listener services (like AbstractRerunHandler, CatalogPartitionHandler, LateRerunHandler,
MetadataMappingService, RetryHandler) will be impacted which depend on JMS notification and
this need to fixed in Falcon. But it does not mean that we should not add other service.

> Generate notification for Falcon job status
> -------------------------------------------
>
>                 Key: FALCON-1313
>                 URL: https://issues.apache.org/jira/browse/FALCON-1313
>             Project: Falcon
>          Issue Type: New Feature
>          Components: common
>    Affects Versions: 0.6.1
>            Reporter: Peeyush Bishnoi
>            Assignee: Peeyush Bishnoi
>         Attachments: Falconjobstatusnotification.pdf
>
>
> Falcon should generate Email based notification for job status to the user/group when
job finish either with success/failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message