falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balu Vellanki" <bvella...@hortonworks.com>
Subject Review Request 38082: Status of scheduled Process entity is shown as submitted in corner case
Date Thu, 03 Sep 2015 00:29:10 GMT

This is an automatically generated e-mail. To reply, visit:

Review request for Falcon, Ajay Yadava, Sowmya Ramesh, and Venkat Ranganathan.

Bugs: FALCON-1371

Repository: falcon-git


Create a process entity with 
- frequency of 1 hour
- validity start as "2012-07-01T00:00Z" 
- and validity end as "2012-07-01T01:02Z"

Now list the process entity after one hour. The process coordinator status in Oozie is "succeeded".
But Falcon shows the status as submitted.


  common/pom.xml 42facc9 
  common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 4d45cc7

  oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 7e6cd6c 
  prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 78964dd 

Diff: https://reviews.apache.org/r/38082/diff/


This patch was tested end2end for the following scenarios.

1. When a bundle completed running successfully, status is now shown as "Succeeded" instead
of "Submitted"

2. When the following feed is submitted, a coordinator is created for replication on backupCluster,
but no coordinator is created on primaryCluster for retention because the feed validity on
primaryCluster is in the past. There was a bug in code which was causing feed status to be
shown as "submitted" if any of the clusters have a MISSING bundle. After this fix, the status
will be shown as running.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="cleansedEmailFeed" description="Cleansed customer emails" xmlns="uri:falcon:feed:0.1">
    <late-arrival cut-off="hours(4)"/>
        <cluster name="primaryCluster" type="source">
            <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
            <retention limit="hours(90)" action="delete"/>
                <location type="data" path="/user/ambari-qa/falcon/demo/primary/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
                <location type="stats" path="/"/>
                <location type="meta" path="/"/>
        <cluster name="backupCluster" type="target">
            <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
            <retention limit="hours(90)" action="delete"/>
                <location type="data" path="/falcon/demo/bcp/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
                <location type="stats" path="/"/>
                <location type="meta" path="/"/>
        <location type="data" path="/user/ambari-qa/falcon/demo/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
        <location type="stats" path="/"/>
        <location type="meta" path="/"/>
    <ACL owner="ambari-qa" group="users" permission="0x755"/>
    <schema location="/none" provider="/none"/>

3. If a bundle is killed using oozie command line, the status of entity will be now shown

4. Added support for FAILED.

5. Added methods to avoid calling findLatestBundle(...) repeatedly.


Balu Vellanki

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message