falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balu Vellanki" <bvella...@hortonworks.com>
Subject Review Request 38082: Status of scheduled Process entity is shown as submitted in corner case
Date Thu, 03 Sep 2015 00:29:10 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38082/
-----------------------------------------------------------

Review request for Falcon, Ajay Yadava, Sowmya Ramesh, and Venkat Ranganathan.


Bugs: FALCON-1371
    https://issues.apache.org/jira/browse/FALCON-1371


Repository: falcon-git


Description
-------

Create a process entity with 
- frequency of 1 hour
- validity start as "2012-07-01T00:00Z" 
- and validity end as "2012-07-01T01:02Z"

Now list the process entity after one hour. The process coordinator status in Oozie is "succeeded".
But Falcon shows the status as submitted.


Diffs
-----

  common/pom.xml 42facc9 
  common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 4d45cc7

  oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 7e6cd6c 
  prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 78964dd 

Diff: https://reviews.apache.org/r/38082/diff/


Testing
-------

This patch was tested end2end for the following scenarios.

1. When a bundle completed running successfully, status is now shown as "Succeeded" instead
of "Submitted"

2. When the following feed is submitted, a coordinator is created for replication on backupCluster,
but no coordinator is created on primaryCluster for retention because the feed validity on
primaryCluster is in the past. There was a bug in code which was causing feed status to be
shown as "submitted" if any of the clusters have a MISSING bundle. After this fix, the status
will be shown as running.
{code}
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="cleansedEmailFeed" description="Cleansed customer emails" xmlns="uri:falcon:feed:0.1">
    <groups>churnAnalysisDataPipeline</groups>
    <frequency>hours(1)</frequency>
    <timezone>UTC</timezone>
    <late-arrival cut-off="hours(4)"/>
    <clusters>
        <cluster name="primaryCluster" type="source">
            <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
            <retention limit="hours(90)" action="delete"/>
            <locations>
                <location type="data" path="/user/ambari-qa/falcon/demo/primary/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
                <location type="stats" path="/"/>
                <location type="meta" path="/"/>
            </locations>
        </cluster>
        <cluster name="backupCluster" type="target">
            <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
            <retention limit="hours(90)" action="delete"/>
            <locations>
                <location type="data" path="/falcon/demo/bcp/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
                <location type="stats" path="/"/>
                <location type="meta" path="/"/>
            </locations>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/user/ambari-qa/falcon/demo/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
        <location type="stats" path="/"/>
        <location type="meta" path="/"/>
    </locations>
    <ACL owner="ambari-qa" group="users" permission="0x755"/>
    <schema location="/none" provider="/none"/>
</feed>
{code}

3. If a bundle is killed using oozie command line, the status of entity will be now shown
as KILLED

4. Added support for FAILED.

5. Added methods to avoid calling findLatestBundle(...) repeatedly.


Thanks,

Balu Vellanki


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message