falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Praveen Adlakha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1565) Listing API non-intuitive response if time > endTime
Date Sat, 19 Dec 2015 09:24:46 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065304#comment-15065304
] 

Praveen Adlakha commented on FALCON-1565:
-----------------------------------------

Hi All,

I have submitted the patch for this one and have done the required testing also.

In distributed mode:
{code}
dataqa@blr-test-129:/usr/local/falcon/falcon-distributed-0.9-SNAPSHOT$ bin/falcon instance
-type feed -name feed1 -start 2026-10-09T18:00Z -listing 
ERROR: Bad Request;local/Specified End date 2015-12-19T07:21Z is before the entity was scheduled
2026-10-09T18:00Z


feed definition:

dataqa@blr-test-129:/usr/local/falcon/falcon-distributed-0.9-SNAPSHOT$ bin/falcon entity -type
feed -name feed1 -definition
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="feed1" description="Input File" xmlns="uri:falcon:feed:0.1">
    <frequency>minutes(5)</frequency>
    <timezone>UTC</timezone>
    <late-arrival cut-off="minutes(2)"/>
    <clusters>
        <cluster name="local" type="source">
            <validity start="2015-10-09T14:00Z" end="2025-10-09T18:00Z"/>
            <retention limit="days(1000000)" action="delete"/>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/user/dataqa/testEffective/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
    </locations>
    <ACL owner="dataqa" group="dataqa" permission="*"/>
    <schema location="none" provider="none"/>
    <properties>
        <property name="field1" value="value1"/>
        <property name="field2" value="value2"/>
    </properties>
</feed>

Process definition 

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<process name="DWH-CI-daily-supply" xmlns="uri:falcon:process:0.1">
    <clusters>
        <cluster name="local">
            <validity start="2015-12-20T01:00Z" end="2015-12-21T00:00Z"/>
        </cluster>
    </clusters>
    <parallel>1</parallel>
    <order>FIFO</order>
    <frequency>days(1)</frequency>
    <timezone>UTC</timezone>
    <inputs>
        <input name="feed1" feed="feed1"  start="today(-1,0)" end="today(-1,0)"/>
    </inputs>
    <properties>
        <property name="queueName" value="supply"/>
        <property name="jobPriority" value="LOW"/>
        <property name="date" value="${formatTime(dateOffset(instanceTime(), -1, 'DAY'),
'yyyy-MM-dd')}"/>
        <property name="dbUploadHostName" value="glgw4006.grid.uh1.inmobi.com"/>
        <property name="dbuploadscript" value="/home/supply/tar//daily/daily-tar-job.sh"/>
        <property name="hdfspath" value="/projects/supply/data"/>
        <property name="tarpath" value="opt/mkhoj/var/sibyl/supply"/>
    </properties>
    <workflow path="/projects/ivory/working/workflow.xml" lib="/projects/ivory/working/lib/"/>
    <retry policy="periodic" delay="minutes(2)" attempts="3"/>
</process>

dataqa@blr-test-129:/usr/local/falcon/falcon-distributed-0.9-SNAPSHOT$ date
Sat Dec 19 13:11:02 IST 2015

dataqa@blr-test-129:/usr/local/falcon/falcon-distributed-0.9-SNAPSHOT$ bin/falcon entity -type
process -name DWH-CI-daily-supply  -status
local/RUNNING

{code}

I am not sure what should be the output in case of process please verify and confirm for the
same.

In Embedded mode:

{code}
dataqa@blr-test-129:/usr/local/falcon/falcon-0.9-SNAPSHOT$ bin/falcon entity -type feed -name
feed1 -definition
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="feed1" description="Input File" xmlns="uri:falcon:feed:0.1">
    <frequency>minutes(5)</frequency>
    <timezone>UTC</timezone>
    <late-arrival cut-off="minutes(2)"/>
    <clusters>
        <cluster name="local" type="source">
            <validity start="2015-10-09T14:00Z" end="2025-10-09T18:00Z"/>
            <retention limit="days(1000000)" action="delete"/>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/user/dataqa/testEffective/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
    </locations>
    <ACL owner="dataqa" group="dataqa" permission="*"/>
    <schema location="none" provider="none"/>
    <properties>
        <property name="field1" value="value1"/>
        <property name="field2" value="value2"/>
    </properties>
</feed>



Process definition 

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<process name="DWH-CI-daily-supply" xmlns="uri:falcon:process:0.1">
    <clusters>
        <cluster name="local">
            <validity start="2015-12-20T01:00Z" end="2015-12-21T00:00Z"/>
        </cluster>
    </clusters>
    <parallel>1</parallel>
    <order>FIFO</order>
    <frequency>days(1)</frequency>
    <timezone>UTC</timezone>
    <inputs>
        <input name="feed1" feed="feed1"  start="today(-1,0)" end="today(-1,0)"/>
    </inputs>
    <properties>
        <property name="queueName" value="supply"/>
        <property name="jobPriority" value="LOW"/>
        <property name="date" value="${formatTime(dateOffset(instanceTime(), -1, 'DAY'),
'yyyy-MM-dd')}"/>
        <property name="dbUploadHostName" value="glgw4006.grid.uh1.inmobi.com"/>
        <property name="dbuploadscript" value="/home/supply/tar//daily/daily-tar-job.sh"/>
        <property name="hdfspath" value="/projects/supply/data"/>
        <property name="tarpath" value="opt/mkhoj/var/sibyl/supply"/>
    </properties>
    <workflow path="/projects/ivory/working/workflow.xml" lib="/projects/ivory/working/lib/"/>
    <retry policy="periodic" delay="minutes(2)" attempts="3"/>
</process>

dataqa@blr-test-129:/usr/local/falcon/falcon-0.9-SNAPSHOT$ date
Sat Dec 19 14:27:57 IST 2015

dataqa@blr-test-129:/usr/local/falcon/falcon-0.9-SNAPSHOT$ bin/falcon entity -type process
-name DWH-CI-daily-supply  -status
default/RUNNING
{code}

> Listing API non-intuitive response if time > endTime
> ----------------------------------------------------
>
>                 Key: FALCON-1565
>                 URL: https://issues.apache.org/jira/browse/FALCON-1565
>             Project: Falcon
>          Issue Type: Bug
>    Affects Versions: 0.8
>         Environment: QA
>            Reporter: Pragya Mittal
>            Assignee: Praveen Adlakha
>              Labels: newbie
>             Fix For: 0.9
>
>         Attachments: FALCON-1565.patch
>
>
> While listing for an entity with time > endTime, response shows stack trace. Instead
error mesage should be shown as response.
> Feed definition is :
> {noformat}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <feed name="sla-feed" description="clicks log" xmlns="uri:falcon:feed:0.1">
>     <partitions>
>         <partition name="country"/>
>         <partition name="colo"/>
>     </partitions>
>     <frequency>minutes(2)</frequency>
>     <sla slaLow="minutes(1)" slaHigh="minutes(3)"/>
>     <timezone>UTC</timezone>
>     <late-arrival cut-off="hours(6)"/>
>     <clusters>
>         <cluster name="A9e7e0672-e02bcc65">
>             <validity start="2015-10-28T11:58Z" end="2015-10-30T11:17Z"/>
>             <retention limit="days(1000000)" action="delete"/>
>         </cluster>
>     </clusters>
>     <locations>
>         <location type="data" path="/tmp/falcon-regression/FeedSlaMonitoring/input/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
>         <location type="stats" path="/projects/falcon/clicksStats"/>
>         <location type="meta" path="/projects/falcon/clicksMetaData"/>
>     </locations>
>     <ACL owner="pragya" group="dataqa" permission="*"/>
>     <schema location="/schema/clicks" provider="protobuf"/>
>     <properties>
>         <property name="field1" value="value1"/>
>     </properties>
> </feed>
> {noformat}
> Listing API response
> {noformat}
> dataqa@lda01:/mnt/users/pragya/defn/sla$ falcon instance -type feed -name sla-feed -start
2015-10-30T11:20Z -listing
> ERROR: Bad Request;<?xml version="1.0" encoding="UTF-8" standalone="yes"?><feedInstanceResult><status>FAILED</status><message>ua1/org.apache.falcon.FalconException::javax.ws.rs.WebApplicationException:
javax.xml.bind.UnmarshalException: unexpected element (uri:&quot;&quot;, local:&quot;instancesResult&quot;).
Expected elements are &lt;{}feedInstanceResult&gt;,&lt;{}instance&gt;,&lt;{}result&gt;
> {noformat}
> Stack trace is :
> {noformat}
> 2015-10-28 12:15:54,414 ERROR - [1963200284@qtp-2030538903-5 - 0f02eeb7-1f02-4bea-bbb7-85d5e61b568f:dataqa:GET//instance/listing/feed/sla-feed]
~ Failed to get instances listing (AbstractInstanceManager:528)
> org.apache.falcon.FalconException: Specified End date 2015-10-28T12:15Z is before the
entity was scheduled 2015-10-30T11:20Z
> 	at org.apache.falcon.resource.AbstractInstanceManager.getStartAndEndDate(AbstractInstanceManager.java:853)
> 	at org.apache.falcon.resource.AbstractInstanceManager.getStartAndEndDate(AbstractInstanceManager.java:842)
> 	at org.apache.falcon.resource.AbstractInstanceManager.getListing(AbstractInstanceManager.java:524)
> 	at org.apache.falcon.resource.InstanceManager.getListing(InstanceManager.java:141)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:497)
> 	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> 	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> 	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message