falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ying Zheng" <yzh...@hortonworks.com>
Subject Re: Review Request 37520: FALCON-1231 Reliable notifications for Oozie workflow status
Date Tue, 29 Sep 2015 00:04:10 GMT


> On Aug. 26, 2015, 5:39 p.m., Ying Zheng wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java,
line 48
> > <https://reviews.apache.org/r/37520/diff/2/?file=1042812#file1042812line48>
> >
> >     Shall this class be renamed to WorkflowJobStatusNotificationService, since it
also notifies start and suspend status now?
> 
> Pallavi Rao wrote:
>     Ying, I had the exact same thought. But, didn't change it because the startup.properties
of existing installations have the following lines:
>     ##### Falcon Services #####
>     *.application.services=org.apache.falcon.security.AuthenticationInitializationService,\
>                             org.apache.falcon.workflow.WorkflowJobEndNotificationService,
\
>     
>     An upgrade would require a change to startup properties by the admin. Didn't want
to break backward compatibility.

Here you don't need to change the key but only the value in startup properties in the source
code (under src/conf/). This can be made as part of the upgrade. Also confirmed this with
Venkat and Sowmya. Venkat suggests that if you are really concerned with backward compatibility,
you can change the name and introduce a dummy class for "WorkflowJobEndNotificationService".


- Ying


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37520/#review96559
-----------------------------------------------------------


On Sept. 23, 2015, 6:10 a.m., Pallavi Rao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37520/
> -----------------------------------------------------------
> 
> (Updated Sept. 23, 2015, 6:10 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1231
>     https://issues.apache.org/jira/browse/FALCON-1231
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Workflow completion is currently dependent on post processing action putting messages
(success/failure) to a JMS queue.
> When failures occur on Hadoop / Oozie, the post processing action may not get executed
either. In such cases, Falcon will be blind to the status of the workflow and re-tries will
not happen.
> Instead, use JMS notifications of Oozie to get job completion notifications.
> 
> The patch is backward compatible. It will handle notifications from post-processing and
Oozie. That way users are not forced to perform any migration steps, but, can move to Oozie
notifications in a phased manner.
> 
> 
> Diffs
> -----
> 
>   client/src/main/java/org/apache/falcon/resource/InstancesResult.java 76bb4b0 
>   common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java d0b09df

>   common/src/main/java/org/apache/falcon/entity/WorkflowNameBuilder.java 6890594 
>   common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java ef9da45

>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java 9456fb9

>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239

>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionListener.java 2d3a477

>   common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java
c4f8843 
>   common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java
0b560bb 
>   common/src/test/java/org/apache/falcon/entity/TestWorkflowNameBuilder.java 6060731

>   common/src/test/java/org/apache/falcon/workflow/WorkflowExecutionContextTest.java 65a057d

>   common/src/test/java/org/apache/falcon/workflow/WorkflowJobEndNotificationServiceTest.java
b7df443 
>   messaging/pom.xml 6528278 
>   messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java d3178fb

>   messaging/src/test/java/org/apache/falcon/messaging/JMSMessageConsumerTest.java 5bbcc87

>   oozie-el-extensions/src/main/conf/oozie-site.xml 0925b41 
>   oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java cff1187 
>   oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 0441f7c

>   oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java
9d31d17 
>   rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java 785dce8 
>   rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java b952bbe 
>   webapp/src/conf/oozie/conf/oozie-site.xml 8545ef9 
> 
> Diff: https://reviews.apache.org/r/37520/diff/
> 
> 
> Testing
> -------
> 
> UTs added. Manual testing done.
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message