falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Samarth Gupta (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FALCON-494) update feature broken
Date Wed, 25 Jun 2014 07:50:24 GMT
Samarth Gupta created FALCON-494:
------------------------------------

             Summary: update feature broken 
                 Key: FALCON-494
                 URL: https://issues.apache.org/jira/browse/FALCON-494
             Project: Falcon
          Issue Type: Bug
          Components: general
    Affects Versions: 0.6
         Environment: QA
            Reporter: Samarth Gupta
            Priority: Blocker
             Fix For: 0.5, 0.4


when we update an process / feed , the update take place from current time, and all the future
instance of the old coordinator are moved to new coord with new definition. 

However with the latest falcon, the coord which is getting created is having all instances
which have have been already processed by old coodinator. Resulting in rerunning on those
instances  which are already in RUNNING / READY / WAITING state in old coord. 

in the logs, following lines can be seen which says the new start time :

{code}
2014-06-25 06:36:40,819 INFO  - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395
1118a58a-4a4e-4052-afea-40d4960470f3] ~ Submitted 0000328-140624044727425-oozie-oozi-B on
cluster corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e with
properties: {nameNode=hdfs://192.168.138.182:8020, oozie.use.system.libpath=true, field2=value2,
jobTracker=192.168.138.182:8021, field1=value1, falcon.libpath=/projectsTest/ivory/working/lib,
colo.name=ua3, oozie.bundle.application.path=${nameNode}/projects/ivory/staging/falcon/workflows/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395/1403678197513,
user.name=dataqa} (OozieWorkflowEngine:1316)
2014-06-25 06:36:40,827 INFO  - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395
1118a58a-4a4e-4052-afea-40d4960470f3] ~ New bundle 0000328-140624044727425-oozie-oozi-B scheduled
successfully with start time 2014-06-25T06:39Z (OozieWorkflowEngine:1131)
2014-06-25 06:36:40,828 DEBUG - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395
1118a58a-4a4e-4052-afea-40d4960470f3] ~ Updating endtime of coord 0000323-140624044727425-oozie-oozi-C
to 2014-06-25T06:39Z on cluster corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e
(OozieWorkflowEngine:1055)
{code}

but in the coord xml the start time is till the original process time :

{code}
2014-06-25 06:36:39,925 DEBUG - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395
1118a58a-4a4e-4052-afea-40d4960470f3] ~ Writing definition to /projects/ivory/staging/falcon/workflows/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395/1403678197513/DEFAULT/coordinator.xml
on cluster corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e
(OozieWorkflowBuilder:338)
2014-06-25 06:36:39,925 DEBUG - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395
1118a58a-4a4e-4052-afea-40d4960470f3] ~ <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<coordinator-app name="FALCON_PROCESS_DEFAULT_agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395"
frequency="${coord:minutes(1)}" start="2014-06-25T06:34Z" end="2014-06-25T06:44Z" timezone="UTC"
xmlns="uri:oozie:coordinator:0.3">
    <controls>
        <timeout>30</timeout>
        <concurrency>1</concurrency>
        <execution>FIFO</execution>
        <throttle>60</throttle>
    </controls>
    <datasets>
        <dataset name="inputData" frequency="${coord:minutes(1)}" initial-instance="2009-02-01T00:00Z"
timezone="UTC">
            <uri-template>hdfs://192.168.138.182:8020/tmp/falcon-regression/NewPrismProcessUpdateTest/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
            <done-flag></done-flag>
        </dataset>
        <dataset name="inputData2" frequency="${coord:minutes(1)}" initial-instance="2009-02-01T00:00Z"
timezone="UTC">
            <uri-template>hdfs://192.168.138.182:8020/tmp/falcon-regression/NewPrismProcessUpdateTest/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
            <done-flag></done-flag>
        </dataset>
{code}

most of the regression NewPrismProcessUpdateTest is failing coz of this.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message