falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1709) FIFO order is not followed when scheduled using native scheduler
Date Mon, 04 Jan 2016 09:44:39 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15080898#comment-15080898
] 

Pallavi Rao commented on FALCON-1709:
-------------------------------------

Thanks [~pavan kumar], [~ajayyadava] for the review!
Committed to master:
   c81db67..20bacfa  master -> master

> FIFO order is not followed when scheduled using native scheduler
> ----------------------------------------------------------------
>
>                 Key: FALCON-1709
>                 URL: https://issues.apache.org/jira/browse/FALCON-1709
>             Project: Falcon
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 0.9
>            Reporter: Pragya Mittal
>            Assignee: Pallavi Rao
>             Fix For: 0.9
>
>         Attachments: FALCON-1709.patch
>
>
> 1. status API as of now shows all the materialized instances along with the status of
launched instances. Same should be reflected in case of native scheduler.
> {noformat}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <process name="ProcessWfTest-agregator-coord16-7b27e3a1" xmlns="uri:falcon:process:0.1">
>     <clusters>
>         <cluster name="ProcessWfTest-corp-e5f9ec42">
>             <validity start="2015-12-30T10:32Z" end="2015-12-30T10:52Z"/>
>         </cluster>
>     </clusters>
>     <parallel>4</parallel>
>     <order>FIFO</order>
>     <frequency>minutes(1)</frequency>
>     <timezone>UTC</timezone>
>     <properties>
>         <property name="queueName" value="default"/>
>         <property name="fileTime" value="${formatTime(dateOffset(instanceTime(), 1,
'DAY'), 'yyyy-MMM-dd')}"/>
>         <property name="user" value="${user()}"/>
>     </properties>
>     <workflow path="/tmp/falcon-regression/ProcessWfTest/aggregator"/>
>     <retry policy="periodic" delay="minutes(3)" attempts="3"/>
>     <ACL owner="pragya" group="dataqa" permission="*"/>
> </process>
> {noformat}
> On hitting status from CLI :
> {noformat}
> dataqa@lda01:~$ fip -name ProcessWfTest-agregator-coord16-7b27e3a1 -status
> Consolidated Status: SUCCEEDED
> Instances:
> Instance		Cluster		SourceCluster		Status		Start		End		Details					Log
> -----------------------------------------------------------------------------------------------
> 2015-12-30T10:32Z	ProcessWfTest-corp-e5f9ec42	-	RUNNING	2015-12-30T10:37Z	2015-12-30T10:37Z
-	http://lda01:11000/oozie?job=0000102-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	RUNNING	http://lda01:8088/proxy/application_1445322981080_3831/
> 2015-12-30T10:33Z	ProcessWfTest-corp-e5f9ec42	-	RUNNING	2015-12-30T10:37Z	2015-12-30T10:37Z
-	http://lda01:11000/oozie?job=0000104-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	RUNNING	http://lda01:8088/proxy/application_1445322981080_3832/
> 2015-12-30T10:34Z	-	-	-	-	-	-	-
> 2015-12-30T10:35Z	-	-	-	-	-	-	-
> 2015-12-30T10:36Z	ProcessWfTest-corp-e5f9ec42	-	RUNNING	2015-12-30T10:37Z	2015-12-30T10:37Z
-	http://lda01:11000/oozie?job=0000108-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	RUNNING	http://lda01:8088/proxy/application_1445322981080_3834/
> 2015-12-30T10:37Z	ProcessWfTest-corp-e5f9ec42	-	RUNNING	2015-12-30T10:37Z	2015-12-30T10:37Z
-	http://lda01:11000/oozie?job=0000106-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	RUNNING	http://lda01:8088/proxy/application_1445322981080_3833/
> Additional Information:
> Response: ua1/STATUS
> Request Id: ua1/362835397@qtp-690339675-11 - c655267d-b177-43df-a2cf-c63f10694c9b
> dataqa@lda01:~$ date
> Wed Dec 30 10:37:49 GMT 2015
> {noformat}
> As we can see instances of time 2015-12-30T10:34Z and 2015-12-30T10:35Z didnt get materialized
while instances after 2015-12-30T10:35Z got launched.
> 2. FIFO order is not preserved.
> When i hit teh status command again from CLI following response was recorded.
> {noformat}
> dataqa@lda01:~$ date
> Wed Dec 30 10:42:29 GMT 2015
> dataqa@lda01:~$ fip -name ProcessWfTest-agregator-coord16-7b27e3a1 -status
> Consolidated Status: SUCCEEDED
> Instances:
> Instance		Cluster		SourceCluster		Status		Start		End		Details					Log
> -----------------------------------------------------------------------------------------------
> 2015-12-30T10:33Z	ProcessWfTest-corp-e5f9ec42	-	KILLED	2015-12-30T10:41Z	2015-12-30T10:42Z
-	http://lda01:11000/oozie?job=0000104-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	FAILED/KILLED	http://lda01:8088/proxy/application_1445322981080_3861/
> 2015-12-30T10:34Z	ProcessWfTest-corp-e5f9ec42	-	RUNNING	2015-12-30T10:42Z	2015-12-30T10:42Z
-	http://lda01:11000/oozie?job=0000110-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	RUNNING	http://lda01:8088/proxy/application_1445322981080_3874/
> 2015-12-30T10:35Z	ProcessWfTest-corp-e5f9ec42	-	RUNNING	2015-12-30T10:42Z	2015-12-30T10:42Z
-	http://lda01:11000/oozie?job=0000112-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	RUNNING	http://lda01:8088/proxy/application_1445322981080_3870/
> 2015-12-30T10:36Z	ProcessWfTest-corp-e5f9ec42	-	KILLED	2015-12-30T10:41Z	2015-12-30T10:42Z
-	http://lda01:11000/oozie?job=0000108-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	FAILED/KILLED	http://lda01:8088/proxy/application_1445322981080_3863/
> 2015-12-30T10:37Z	ProcessWfTest-corp-e5f9ec42	-	KILLED	2015-12-30T10:41Z	2015-12-30T10:42Z
-	http://lda01:11000/oozie?job=0000106-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	FAILED/KILLED	http://lda01:8088/proxy/application_1445322981080_3862/
> 2015-12-30T10:38Z	ProcessWfTest-corp-e5f9ec42	-	RUNNING	2015-12-30T10:42Z	2015-12-30T10:42Z
-	http://lda01:11000/oozie?job=0000114-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	RUNNING	http://lda01:8088/proxy/application_1445322981080_3873/
> 2015-12-30T10:39Z	ProcessWfTest-corp-e5f9ec42	-	KILLED	2015-12-30T10:39Z	2015-12-30T10:39Z
-	http://lda01:11000/oozie?job=0000116-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	FAILED/KILLED	http://lda01:8088/proxy/application_1445322981080_3852/
> 2015-12-30T10:40Z	ProcessWfTest-corp-e5f9ec42	-	KILLED	2015-12-30T10:40Z	2015-12-30T10:40Z
-	http://lda01:11000/oozie?job=0000118-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	FAILED/KILLED	http://lda01:8088/proxy/application_1445322981080_3855/
> 2015-12-30T10:41Z	ProcessWfTest-corp-e5f9ec42	-	KILLED	2015-12-30T10:41Z	2015-12-30T10:41Z
-	http://lda01:11000/oozie?job=0000120-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	FAILED/KILLED	http://lda01:8088/proxy/application_1445322981080_3858/
> 2015-12-30T10:42Z	ProcessWfTest-corp-e5f9ec42	-	RUNNING	2015-12-30T10:42Z	2015-12-30T10:42Z
-	http://lda01:11000/oozie?job=0000122-151230090106575-oozie-oozi-W
> actions:
>  hdfscommands	OK	-
>  aggregator	RUNNING	http://lda01:8088/proxy/application_1445322981080_3869/
> Additional Information:
> Response: ua1/STATUS
> Request Id: ua1/397575720@qtp-690339675-12 - d0942fed-ec3a-40a6-86e5-f0dfac6564c2
> {noformat}
> Instances of time 2015-12-30T10:34Z, 2015-12-30T10:35Z and 2015-12-30T10:38Z are running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message