Mailing-List: contact dev-help@falcon.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.apache.org
Date: Tue, 22 Dec 2015 10:59:46 +0000 (UTC)
From: "Srikanth Sundarrajan (JIRA)" <jira@apache.org>
To: dev@falcon.incubator.apache.org
Message-ID: <JIRA.12923444.1450708904000.92139.1450781986793@Atlassian.JIRA>
In-Reply-To: <JIRA.12923444.1450708904000@Atlassian.JIRA>
References: <JIRA.12923444.1450708904000@Atlassian.JIRA>
 <JIRA.12923444.1450708904523@arcas>
Subject: [jira] [Commented] (FALCON-1686) Support for reprocessing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/FALCON-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067956#comment-15067956 ] 

Srikanth Sundarrajan commented on FALCON-1686:
----------------------------------------------

This is the classic backfill issue. Not sure if the effective time would solve it. I would think that the effective time feature would require the time to be between the start time and end time of the process definition. We have had this issue as well in the past and solution we had used was the same as suggested by [~massdosage@gmail.com] and isn't ideal. Unless there is a hole in the data, simple update of the process with a earlier start date should be supported.

> Support for reprocessing
> ------------------------
>
>                 Key: FALCON-1686
>                 URL: https://issues.apache.org/jira/browse/FALCON-1686
>             Project: Falcon
>          Issue Type: Improvement
>    Affects Versions: 0.7
>            Reporter: Mass Dosage
>
> We have a number of ETL jobs which we schedule to run on a regular basis with Falcon. This works fine. However, we often have cases where we need to run the exact same jobs over past date ranges in order to reprocess data after a code change. There doesn't seem to be any easy way to do this in Falcon at the moment. Ideally we'd have a controlled way of saying "run this process for dates between X and Y". There should also be a way to control whether downstream processes are triggered by the data being reprocessed or not. In some cases you may want downstream jobs to also run on the new data but in other cases you might not. 
> With Oozie, if one wants to reprocess data from any time in history, one can update the start & end-dates (using the job.properties file) and submit a new coordinator to run alongside the existing one. As the coordinator-ids are unique they do not clash. In Falcon, processes are defined by their readable name so one would need to update that in the process file directly. 
> We are currently working around this issue by making a copy of the original Falcon process, giving it a different name and changing the dates. This isn't ideal and leads to a lot of XML duplication. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)