Return-Path: 
 <dev-return-12646-apmail-falcon-dev-archive=falcon.apache.org@falcon.apache.org>
X-Original-To: apmail-falcon-dev-archive@minotaur.apache.org
Delivered-To: apmail-falcon-dev-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5711B18C0B
	for <apmail-falcon-dev-archive@minotaur.apache.org>;
 Mon, 15 Jun 2015 08:45:29 +0000 (UTC)
Received: (qmail 97859 invoked by uid 500); 15 Jun 2015 08:45:28 -0000
Delivered-To: apmail-falcon-dev-archive@falcon.apache.org
Received: (qmail 97820 invoked by uid 500); 15 Jun 2015 08:45:28 -0000
Mailing-List: contact dev-help@falcon.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@falcon.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@falcon.apache.org>
List-Post: <mailto:dev@falcon.apache.org>
List-Id: <dev.falcon.apache.org>
Reply-To: dev@falcon.apache.org
Delivered-To: mailing list dev@falcon.apache.org
Received: (qmail 97800 invoked by uid 99); 15 Jun 2015 08:45:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jun 2015 08:45:28 +0000
X-ASF-Spam-Status: No, hits=-2000.4 required=5.0
	tests=ALL_TRUSTED,RP_MATCHES_RCVD,WEIRD_PORT
X-Spam-Check-By: apache.org
Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 15 Jun 2015 08:43:16 +0000
Received: (qmail 97734 invoked by uid 99); 15 Jun 2015 08:45:02 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jun 2015 08:45:02 +0000
Date: Mon, 15 Jun 2015 08:45:02 +0000 (UTC)
From: "Ajay Yadava (JIRA)" <jira@apache.org>
To: dev@falcon.incubator.apache.org
Message-ID: <JIRA.12836070.1433746853000.70234.1434357902407@Atlassian.JIRA>
In-Reply-To: <JIRA.12836070.1433746853000@Atlassian.JIRA>
References: <JIRA.12836070.1433746853000@Atlassian.JIRA>
 <JIRA.12836070.1433746853103@arcas>
Subject: [jira] [Updated] (FALCON-1260) Instance dependency API produces
 incorrect results
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/FALCON-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ajay Yadava updated FALCON-1260:
--------------------------------
    Attachment: FALCON-1260.patch

This patch fixes and adds unit tests for all the issues reported here and more.

> Instance dependency API produces incorrect results
> --------------------------------------------------
>
>                 Key: FALCON-1260
>                 URL: https://issues.apache.org/jira/browse/FALCON-1260
>             Project: Falcon
>          Issue Type: Bug
>          Components: feed, process
>         Environment: QA
>            Reporter: Pragya Mittal
>            Assignee: Ajay Yadava
>             Fix For: 0.7
>
>         Attachments: FALCON-1260.patch
>
>
> I have the following definitions :
> Process :
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <process name="InstanceDependencyTest--agregator-coord16-4adb02c6" xmlns="uri:falcon:process:0.1">
> <clusters>
> <cluster name="InstanceDependencyTest--corp-a5b30313">
> <validity start="2015-06-06T09:37Z" end="2015-06-06T10:37Z"/>
> </cluster>
> </clusters>
> <parallel>5</parallel>
> <order>FIFO</order>
> <frequency>minutes(10)</frequency>
> <timezone>UTC</timezone>
> <inputs>
> <input name="inputData" feed="InstanceDependencyTest--raaw-logs16-38bc46f1" start="now(0,-20)" end="now(0,0)"/>
> </inputs>
> <outputs>
> <output name="outputData" feed="InstanceDependencyTest--agregated-logs16-e6cfefc5" instance="now(0,0)"/>
> </outputs>
> <properties>
> <property name="queueName" value="default"/>
> <property name="fileTime" value="${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}"/>
> <property name="user" value="${user()}"/>
> </properties>
> <workflow path="/tmp/falcon-regression/InstanceDependencyTest/aggregator"/>
> <retry policy="periodic" delay="minutes(3)" attempts="3"/>
> <ACL owner="pragya" group="dataqa" permission="*"/>
> </process>
> {code}
> Input feed :
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <feed name="InstanceDependencyTest--raaw-logs16-38bc46f1" description="clicks log" xmlns="uri:falcon:feed:0.1">
> <partitions>
> <partition name="country"/>
> <partition name="colo"/>
> </partitions>
> <frequency>minutes(5)</frequency>
> <timezone>UTC</timezone>
> <late-arrival cut-off="hours(6)"/>
> <clusters>
> <cluster name="InstanceDependencyTest--corp-a5b30313" type="source">
> <validity start="2009-02-01T00:00Z" end="2099-05-01T00:00Z"/>
> <retention limit="months(9000)" action="delete"/>
> </cluster>
> </clusters>
> <locations>
> <location type="data" path="/tmp/falcon-regression/InstanceDependencyTest/input/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
> <location type="stats" path="/projects/falcon/clicksStats"/>
> <location type="meta" path="/projects/falcon/clicksMetaData"/>
> </locations>
> <ACL owner="pragya" group="dataqa" permission="*"/>
> <schema location="/schema/clicks" provider="protobuf"/>
> <properties>
> <property name="field1" value="value1"/>
> <property name="field2" value="value2"/>
> </properties>
> </feed>
> {code}
> Output feed:
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <feed name="InstanceDependencyTest--agregated-logs16-e6cfefc5" description="clicks log" xmlns="uri:falcon:feed:0.1">
> <frequency>minutes(5)</frequency>
> <timezone>UTC</timezone>
> <late-arrival cut-off="hours(6)"/>
> <clusters>
> <cluster name="InstanceDependencyTest--corp-a5b30313" type="source">
> <validity start="2009-02-01T01:00Z" end="2099-05-01T00:00Z"/>
> <retention limit="hours(6)" action="delete"/>
> </cluster>
> </clusters>
> <locations>
> <location type="data" path="/tmp/falcon-regression/InstanceDependencyTest/output-data/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
> <location type="stats" path="/projects/falcon/clicksStats"/>
> <location type="meta" path="/projects/falcon/clicksMetaData"/>
> </locations>
> <ACL owner="pragya" group="dataqa" permission="*"/>
> <schema location="/schema/clicks" provider="protobuf"/>
> <properties>
> <property name="field1" value="value1"/>
> <property name="field2" value="value2"/>
> </properties>
> </feed>
> {code}
> Cluster :
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <cluster name="InstanceDependencyTest--corp-a5b30313" description="" colo="ua1" xmlns="uri:falcon:cluster:0.1">
> <interfaces>
> <interface type="readonly" endpoint="hdfs://192.168.138.200:8020" version="0.20.2"/>
> <interface type="write" endpoint="hdfs://192.168.138.200:8020" version="0.20.2"/>
> <interface type="execute" endpoint="192.168.138.200:8032" version="0.20.2"/>
> <interface type="workflow" endpoint="http://192.168.138.200:11000/oozie/" version="3.1.0"/>
> <interface type="messaging" endpoint="tcp://192.168.138.200:61616?daemon=true" version="5.1.6"/>
> </interfaces>
> <locations>
> <location name="staging" path="/tmp/falcon-regression-staging"/>
> <location name="working" path="/tmp/falcon-regression-working"/>
> <location name="temp" path="/tmp"/>
> </locations>
> <ACL owner="pragya" group="dataqa" permission="*"/>
> <properties>
> <property name="field1" value="value1"/>
> <property name="field2" value="value2"/>
> </properties>
> </cluster>
> {code}
> Usage :
> 1. Output feed :
> {code}
> dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--agregated-logs16-e6cfefc5 -instanceTime 2015-06-06T09:40Z
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:40Z, tag: Output
> Response: ua1/Success!
> Request Id: ua1/1610525991@qtp-1766505436-0 - deabb3d7-fc7e-474a-b315-50da006ebf7a
> {code}
> Although process instance doesnot exist, it says it exists
> 2.InstanceTime in process validaity range but instance doesnt exist
> Input feed:
> {code}
> dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--raaw-logs16-38bc46f1 -instanceTime 2015-06-06T09:40Z
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:56Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:45Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:42Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:49Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:52Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:50Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:57Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:54Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:55Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:53Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:43Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:41Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:51Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:48Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:47Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:46Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:44Z, tag: Input
> Response: ua1/Success!
> Request Id: ua1/1610525991@qtp-1766505436-0 - 8b49d24a-1654-423c-a591-68de039a41f0
> {code}
> Output feed :
> {code}
> dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--agregated-logs16-e6cfefc5 -instanceTime 2015-06-06T09:40Z
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:40Z, tag: Output
> Response: ua1/Success!
> Request Id: ua1/1610525991@qtp-1766505436-0 - c88476ad-0bff-44f4-b43d-82b846e9b1cf
> {code}
> 3. InstanceTime < process start time
> Input feed:
> {code}
> dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--raaw-logs16-38bc46f1 -instanceTime 2014-06-06T08:40Z
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:42Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:43Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:52Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:41Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:45Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:57Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:54Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:44Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:47Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:49Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:46Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:56Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:55Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:48Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:53Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:50Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:51Z, tag: Input
> Response: ua1/Success!
> Request Id: ua1/1610525991@qtp-1766505436-0 - 7e39a727-651a-4efb-925b-2a65aa8b6b
> {code}
> Output feed:
> {code}
> dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--agregated-logs16-e6cfefc5 -instanceTime 2014-06-06T08:40Z
> ERROR: Bad Request;<?xml version="1.0" encoding="UTF-8" standalone="yes"?><dependents><status>FAILED</status><message>ua1/org.apache.falcon.FalconException::javax.ws.rs.WebApplicationException: javax.xml.bind.UnmarshalException: unexpected element (uri:&quot;&quot;, local:&quot;instancesResult&quot;). Expected elements are &lt;{}dependents&gt;,&lt;{}result&gt;
> </message><requestId>ua1/339141107@qtp-212921632-26 - 998c331e-188a-4ca3-88da-bd1a83bc40bb
> </requestId></dependents>
> {code}
> 4. InstanceTime > process end time
> Input feed:
> {code}
> dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--raaw-logs16-38bc46f1 -instanceTime 2016-06-06T08:40Z
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:56Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:49Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:51Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:48Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:55Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:46Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:50Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:53Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:45Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:43Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:42Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:41Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:52Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:57Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:54Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:47Z, tag: Input
> name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:44Z, tag: Input
> Response: ua1/Success!
> Request Id: ua1/1610525991@qtp-1766505436-0 - 7cc314ab-9b3a-413d-9024-eeb5804a88db
> {code}
> Output feed :
> {code}
> dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--agregated-logs16-e6cfefc5 -instanceTime 2016-06-06T08:40Z
> ERROR: Bad Request;<?xml version="1.0" encoding="UTF-8" standalone="yes"?><dependents><status>FAILED</status><message>ua1/org.apache.falcon.FalconException::javax.ws.rs.WebApplicationException: javax.xml.bind.UnmarshalException: unexpected element (uri:&quot;&quot;, local:&quot;instancesResult&quot;). Expected elements are &lt;{}dependents&gt;,&lt;{}result&gt;
> </message><requestId>ua1/339141107@qtp-212921632-26 - c98b92b3-a34a-4f57-97c7-418727c338ab
> </requestId></dependents>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)