Mailing-List: contact dev-help@falcon.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.apache.org
Date: Mon, 11 Jan 2016 04:26:39 +0000 (UTC)
From: "Pallavi Rao (JIRA)" <jira@apache.org>
To: dev@falcon.incubator.apache.org
Message-ID: <JIRA.12928633.1452258130000.70110.1452486399980@Atlassian.JIRA>
In-Reply-To: <JIRA.12928633.1452258130000@Atlassian.JIRA>
References: <JIRA.12928633.1452258130000@Atlassian.JIRA>
 <JIRA.12928633.1452258130429@arcas>
Subject: [jira] [Commented] (FALCON-1741) Rerun API behaviour different in
 case of succeeded instances
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/FALCON-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15091415#comment-15091415 ] 

Pallavi Rao commented on FALCON-1741:
-------------------------------------

When we do a rerun for Oozie, we directly rerun a workflow. Hence you see that it has moved to running immediately. This should in fact be changed to rerun of a co-ordinator action and you might see it moved to READY here too.

When we do a rerun on native scheduler, we schedule the workflow again. This is done intentionally so as not to violate the number of instances running in parallel. For example, if parallelism is 3 and 3 instances are in RUNNING, re-run of 2 older instances immediately, will cause 5 instances to run.

We should mask the wfParams, though.

> Rerun API behaviour different in case of succeeded instances
> ------------------------------------------------------------
>
>                 Key: FALCON-1741
>                 URL: https://issues.apache.org/jira/browse/FALCON-1741
>             Project: Falcon
>          Issue Type: Sub-task
>          Components: oozie, scheduler
>    Affects Versions: 0.9
>            Reporter: Pragya Mittal
>
> In case of succeeded instances , rerun without -force API has different behaviour based on the scheduler choice.
> For oozie :
> The state of instances changes to RUNNING.
> {noformat}
> dataqa@lda01:~$ fiprerun processMerlinOozie -start 2016-01-08T12:13Z -end 2016-01-08T12:15Z
> Consolidated Status: SUCCEEDED
> Instances:
> Instance		Cluster		SourceCluster		Status		Start		End		Details					Log
> -----------------------------------------------------------------------------------------------
> 2016-01-08T12:13Z	ProcessMultipleClustersTest-corp-9706f068	-	RUNNING	2016-01-08T13:03Z	2016-01-08T13:03Z	-	http://8RPCG32.corp.inmobi.com:11000/oozie?job=0001811-160104160825636-oozie-oozi-W
> 2016-01-08T12:13Z	ProcessMultipleClustersTest-corp-0b270a1d	-	RUNNING	2016-01-08T13:03Z	2016-01-08T13:03Z	-	http://lda01:11000/oozie?job=0002247-160104115615658-oozie-oozi-W
> Additional Information:
> Response: ua1/RERUN
> ua2/RERUN
> Request Id: ua1/871377866@qtp-630572412-35 - 7190c4c8-bacb-4639-8d48-c9e639f544da
> ua2/1554129706@qtp-536122141-13 - bc18127b-1bf8-4ea1-99e6-b1f10ba3a441
> {noformat}
> For native :
> The state of instances changes to READY. Also params are present as a part of rerun API>
> {noformat}
> dataqa@lda01:~$ fiprerun ProcessMultipleClustersTest-agregator-coord16-8f55f59b -start 2016-01-08T12:13Z -end 2016-01-08T12:15Z
> Consolidated Status: SUCCEEDED
> Instances:
> Instance		Cluster		SourceCluster		Status		Start		End		Details					Log
> -----------------------------------------------------------------------------------------------
> 2016-01-08T12:13Z	ProcessMultipleClustersTest-corp-9706f068	-	READY	2016-01-08T13:03Z	2016-01-08T13:03Z	-	http://8RPCG32.corp.inmobi.com:11000/oozie?job=0001812-160104160825636-oozie-oozi-W
> Workflow params
> shouldRecord=false
> timeStamp=2016-01-08-12-13
> mapreduce.job.user.name=dataqa
> falconInPaths=NONE
> feedNames=NONE
> oozie.wf.rerun.failnodes=true
> falcon.libpath=/tmp/falcon-regression-working/lib
> ENTITY_PATH=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/3cd7594ed13b218460df53a9e827eb28_1452252282546
> entityType=PROCESS
> systemJMSNotificationEnabled=false
> nominalTime=2016-01-08-12-13
> feedInstancePaths=NONE
> field2=value2
> logDir=hdfs://10.14.114.202:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/logs
> userWorkflowEngine=oozie
> oozie.use.system.libpath=true
> userJMSNotificationEnabled=true
> queueName=default
> falconDataOperation=GENERATE
> falcon.system.rerun=true
> fileTime=${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}
> field1=value1
> workflowEngineUrl=http://10.14.114.202:11000/oozie/
> datasource=NA
> brokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
> userBrokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
> brokerUrl=tcp://localhost:61616
> ENTITY_NAME=FALCON_PROCESS_DEFAULT_ProcessMultipleClustersTest-agregator-coord16-8f55f59b
> availabilityFlag=NA
> userWorkflowName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b-workflow
> brokerTTL=4320
> entityName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b
> srcClusterName=NA
> userBrokerUrl=tcp://10.14.114.202:61616?daemon=true
> falconInputFeeds=NONE
> userWorkflowVersion=1.0
> oozie.libpath=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/3cd7594ed13b218460df53a9e827eb28_1452252282546/lib
> cluster=ProcessMultipleClustersTest-corp-9706f068
> jobPriority=NORMAL
> colo.name=ua1
> nameNode=hdfs://10.14.114.202:8020
> user=${user()}
> jobTracker=10.14.114.202:8021
> oozie.wf.application.path=hdfs://10.14.114.202:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/3cd7594ed13b218460df53a9e827eb28_1452252282546
> 2016-01-08T12:13Z	ProcessMultipleClustersTest-corp-0b270a1d	-	READY	2016-01-08T13:03Z	2016-01-08T13:03Z	-	http://lda01:11000/oozie?job=0002248-160104115615658-oozie-oozi-W
> Workflow params
> fileTime=${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}
> cluster=ProcessMultipleClustersTest-corp-0b270a1d
> availabilityFlag=NA
> falconDataOperation=GENERATE
> falconInputFeeds=NONE
> feedNames=NONE
> colo.name=ua2
> shouldRecord=false
> falcon.libpath=/tmp/falcon-regression-working/lib
> mapreduce.job.user.name=dataqa
> feedInstancePaths=NONE
> systemJMSNotificationEnabled=false
> brokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
> userBrokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
> entityName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b
> falcon.system.rerun=true
> jobTracker=192.168.138.236:8021
> userWorkflowVersion=1.0
> workflowEngineUrl=http://192.168.138.236:11000/oozie/
> jobPriority=NORMAL
> ENTITY_NAME=FALCON_PROCESS_DEFAULT_ProcessMultipleClustersTest-agregator-coord16-8f55f59b
> nameNode=hdfs://192.168.138.236:8020
> nominalTime=2016-01-08-12-13
> oozie.libpath=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/e6720cfbbe1d8a50488a7c47da94b506_1452252285506/lib
> oozie.use.system.libpath=true
> entityType=PROCESS
> srcClusterName=NA
> userWorkflowEngine=oozie
> oozie.wf.rerun.failnodes=true
> timeStamp=2016-01-08-12-13
> brokerUrl=tcp://localhost:61616
> field1=value1
> oozie.wf.application.path=hdfs://192.168.138.236:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/e6720cfbbe1d8a50488a7c47da94b506_1452252285506
> queueName=default
> brokerTTL=4320
> userWorkflowName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b-workflow
> datasource=NA
> ENTITY_PATH=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/e6720cfbbe1d8a50488a7c47da94b506_1452252285506
> userJMSNotificationEnabled=true
> userBrokerUrl=tcp://192.168.138.236:61616?daemon=true
> falconInPaths=NONE
> user=${user()}
> field2=value2
> logDir=hdfs://192.168.138.236:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/logs
> Additional Information:
> Response: ua1/RERUN
> ua2/RERUN
> Request Id: ua1/871377866@qtp-630572412-35 - 8d118d4d-c0ef-4335-a9af-10364498ec4f
> ua2/1554129706@qtp-536122141-13 - c2a3fc50-8b05-47ce-9c85-ca432b96d923
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)