falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao" <pallavi....@inmobi.com>
Subject Re: Review Request 37520: FALCON-1231 Reliable notifications for Oozie workflow status
Date Tue, 29 Sep 2015 08:28:15 GMT


> On Sept. 29, 2015, 12:20 a.m., Ying Zheng wrote:
> > messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java, line
225
> > <https://reviews.apache.org/r/37520/diff/3/?file=1082055#file1082055line225>
> >
> >     As I experimented, coordinator actions can capture all status changes including
the ones provided by workflow jobs. The notification from WORKFLOW_JOB doesn't seem necessary
here.

The WORKFLOW_JOB is given preference and is used for 2 reasons:
1. Doesn't need a second call to Oozie to get the workflow id and corresponding conf.
2. When we move to native scheduler, Oozie coordinator is not going to be used. So, keeping
the patch a little future looking.


- Pallavi


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37520/#review100904
-----------------------------------------------------------


On Sept. 23, 2015, 6:10 a.m., Pallavi Rao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37520/
> -----------------------------------------------------------
> 
> (Updated Sept. 23, 2015, 6:10 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1231
>     https://issues.apache.org/jira/browse/FALCON-1231
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Workflow completion is currently dependent on post processing action putting messages
(success/failure) to a JMS queue.
> When failures occur on Hadoop / Oozie, the post processing action may not get executed
either. In such cases, Falcon will be blind to the status of the workflow and re-tries will
not happen.
> Instead, use JMS notifications of Oozie to get job completion notifications.
> 
> The patch is backward compatible. It will handle notifications from post-processing and
Oozie. That way users are not forced to perform any migration steps, but, can move to Oozie
notifications in a phased manner.
> 
> 
> Diffs
> -----
> 
>   client/src/main/java/org/apache/falcon/resource/InstancesResult.java 76bb4b0 
>   common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java d0b09df

>   common/src/main/java/org/apache/falcon/entity/WorkflowNameBuilder.java 6890594 
>   common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java ef9da45

>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java 9456fb9

>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239

>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionListener.java 2d3a477

>   common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java
c4f8843 
>   common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java
0b560bb 
>   common/src/test/java/org/apache/falcon/entity/TestWorkflowNameBuilder.java 6060731

>   common/src/test/java/org/apache/falcon/workflow/WorkflowExecutionContextTest.java 65a057d

>   common/src/test/java/org/apache/falcon/workflow/WorkflowJobEndNotificationServiceTest.java
b7df443 
>   messaging/pom.xml 6528278 
>   messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java d3178fb

>   messaging/src/test/java/org/apache/falcon/messaging/JMSMessageConsumerTest.java 5bbcc87

>   oozie-el-extensions/src/main/conf/oozie-site.xml 0925b41 
>   oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java cff1187 
>   oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 0441f7c

>   oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java
9d31d17 
>   rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java 785dce8 
>   rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java b952bbe 
>   webapp/src/conf/oozie/conf/oozie-site.xml 8545ef9 
> 
> Diff: https://reviews.apache.org/r/37520/diff/
> 
> 
> Testing
> -------
> 
> UTs added. Manual testing done.
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message