Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.incubator.apache.org
Date: Tue, 8 Jul 2014 08:35:40 +0000 (UTC)
From: "Shwetha G S (JIRA)" <jira@apache.org>
To: dev@falcon.incubator.apache.org
Message-ID: <JIRA.12723576.1403682616674.18353.1404808540155@arcas>
In-Reply-To: <JIRA.12723576.1403682616674@arcas>
References: <JIRA.12723576.1403682616674@arcas>
Subject: [jira] [Updated] (FALCON-494) update feature broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/FALCON-494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shwetha G S updated FALCON-494:
-------------------------------

    Attachment: FALCON-494.patch

> update feature broken 
> ----------------------
>
>                 Key: FALCON-494
>                 URL: https://issues.apache.org/jira/browse/FALCON-494
>             Project: Falcon
>          Issue Type: Bug
>          Components: general
>    Affects Versions: 0.6
>         Environment: QA
>            Reporter: Samarth Gupta
>            Assignee: Suhas Vasu
>            Priority: Blocker
>             Fix For: 0.4, 0.5
>
>         Attachments: FALCON-494.patch
>
>
> when we update an process / feed , the update take place from current time, and all the future instance of the old coordinator are moved to new coord with new definition. 
> However with the latest falcon, the coord which is getting created is having all instances which have have been already processed by old coodinator. Resulting in rerunning on those instances  which are already in RUNNING / READY / WAITING state in old coord. 
> in the logs, following lines can be seen which says the new start time :
> {code}
> 2014-06-25 06:36:40,819 INFO  - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395 1118a58a-4a4e-4052-afea-40d4960470f3] ~ Submitted 0000328-140624044727425-oozie-oozi-B on cluster corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e with properties: {nameNode=hdfs://192.168.138.182:8020, oozie.use.system.libpath=true, field2=value2, jobTracker=192.168.138.182:8021, field1=value1, falcon.libpath=/projectsTest/ivory/working/lib, colo.name=ua3, oozie.bundle.application.path=${nameNode}/projects/ivory/staging/falcon/workflows/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395/1403678197513, user.name=dataqa} (OozieWorkflowEngine:1316)
> 2014-06-25 06:36:40,827 INFO  - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395 1118a58a-4a4e-4052-afea-40d4960470f3] ~ New bundle 0000328-140624044727425-oozie-oozi-B scheduled successfully with start time 2014-06-25T06:39Z (OozieWorkflowEngine:1131)
> 2014-06-25 06:36:40,828 DEBUG - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395 1118a58a-4a4e-4052-afea-40d4960470f3] ~ Updating endtime of coord 0000323-140624044727425-oozie-oozi-C to 2014-06-25T06:39Z on cluster corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e (OozieWorkflowEngine:1055)
> {code}
> but in the coord xml the start time is till the original process time :
> {code}
> 2014-06-25 06:36:39,925 DEBUG - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395 1118a58a-4a4e-4052-afea-40d4960470f3] ~ Writing definition to /projects/ivory/staging/falcon/workflows/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395/1403678197513/DEFAULT/coordinator.xml on cluster corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e (OozieWorkflowBuilder:338)
> 2014-06-25 06:36:39,925 DEBUG - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395 1118a58a-4a4e-4052-afea-40d4960470f3] ~ <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <coordinator-app name="FALCON_PROCESS_DEFAULT_agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395" frequency="${coord:minutes(1)}" start="2014-06-25T06:34Z" end="2014-06-25T06:44Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.3">
>     <controls>
>         <timeout>30</timeout>
>         <concurrency>1</concurrency>
>         <execution>FIFO</execution>
>         <throttle>60</throttle>
>     </controls>
>     <datasets>
>         <dataset name="inputData" frequency="${coord:minutes(1)}" initial-instance="2009-02-01T00:00Z" timezone="UTC">
>             <uri-template>hdfs://192.168.138.182:8020/tmp/falcon-regression/NewPrismProcessUpdateTest/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
>             <done-flag></done-flag>
>         </dataset>
>         <dataset name="inputData2" frequency="${coord:minutes(1)}" initial-instance="2009-02-01T00:00Z" timezone="UTC">
>             <uri-template>hdfs://192.168.138.182:8020/tmp/falcon-regression/NewPrismProcessUpdateTest/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
>             <done-flag></done-flag>
>         </dataset>
> {code}
> most of the regression NewPrismProcessUpdateTest is failing coz of this.


--
This message was sent by Atlassian JIRA
(v6.2#6252)