falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FALCON-1313) Generate notification for Falcon job status
Date Wed, 29 Jul 2015 09:30:06 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645775#comment-14645775
] 

Ajay Yadava edited comment on FALCON-1313 at 7/29/15 9:29 AM:
--------------------------------------------------------------

I see that the new doc mentions the retry scenario, thanks for including it. I have some basic
questions around the requirements/ use cases for alerting?
1. What is the purpose of this feature? The doc says
{quote}
Currently Falcon don’t generate email notification for job status once it finish. Due to
this users need to manually check the job status periodically for success/failure
{quote}
What defines job here? 

{quote}
we want to have functionality to generate email notification once job finishes.
{quote}
What happens if the job doesn't finish? e.g. what if the job is running since long time or
has gone in suspended state?

2. 
{quote}
Currently Falcon don’t generate email notification for job status once it finish. Due to
this users need to manually check the job status periodically for success/failure.
{quote}
Apart from the problem of being generated only when the job finishes, consumers being notified
is only one use case where we need such notifications. Consider the following use case:
There is a pipelines of processes which run, the end result of the pipeline is the update
of hourly reports in a database which powers user dashboards. Pipeline usually takes 30 minutes
to finish and we have kept half an hour of buffer for recovering from any issues and conveyed
to users that the data will be available with a lag of one hour. Let's say there is a problem
in one of the processes in the beginning of the pipeline and it took one hour more than the
usual. Now in this case since the job finished there will be an alert email but that email
is not helpful any longer as we have lost the entire buffer window and have failed to meet
our contract. Current design doesn't answer that.




was (Author: ajayyadava):
I see that the new doc mentions the retry scenario, thanks for including it. I have some basic
questions around the requirements/ use cases for alerting?
1. What is the purpose of this feature? The doc says
{quote}
Currently Falcon don’t generate email notification for job status once it finish. Due to
this users need to manually check the job status periodically for success/failure
{quote}
What defines job here? Is it feed and processes both? 

{quote}
we want to have functionality to generate email notification once job finishes.
{quote}
What happens if the job doesn't finish? e.g. what if the job is running since long time or
has gone in suspended state?

2. 
{quote}
Currently Falcon don’t generate email notification for job status once it finish. Due to
this users need to manually check the job status periodically for success/failure.
{quote}
Apart from the problem of being generated only when the job finishes, consumers being notified
is only one use case where we need such notifications. Consider the following use case:
There is a pipelines of processes which run, the end result of the pipeline is the update
of hourly reports in a database which powers user dashboards. Pipeline usually takes 30 minutes
to finish and we have kept half an hour of buffer for recovering from any issues and conveyed
to users that the data will be available with a lag of one hour. Let's say there is a problem
in one of the processes in the beginning of the pipeline and it took one hour more than the
usual. Now in this case since the job finished there will be an alert email but that email
is not helpful any longer as we have lost the entire buffer window and have failed to meet
our contract. Current design doesn't answer that.



> Generate notification for Falcon job status
> -------------------------------------------
>
>                 Key: FALCON-1313
>                 URL: https://issues.apache.org/jira/browse/FALCON-1313
>             Project: Falcon
>          Issue Type: New Feature
>          Components: common
>    Affects Versions: 0.6.1
>            Reporter: Peeyush Bishnoi
>            Assignee: Peeyush Bishnoi
>         Attachments: Falconjobstatusnotification-1.pdf, Falconjobstatusnotification.pdf
>
>
> Falcon should generate Email based notification for job status to the user/group when
job finish either with success/failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message