falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1406) Effective time in Entity updates.
Date Wed, 23 Nov 2016 03:55:58 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688823#comment-15688823

Srikanth Sundarrajan commented on FALCON-1406:

[~ajayyadava], This is a much cleaner way of retaining history than the current scheme. What
happens in the current scheme is that users simply delete the old entity, create new one with
the same name with a different time range. This change would allow us to track history of
changes for more cleanly. Also when you go back and run an older instance (applicable on old
data), corresponding version of the app is used to process the data. Cleanup of entities isn't
really possible through a tool because no one is going to declare (there isn't a mechanism
either) to say this is a new processing entity created for fixing issues  (backfills, code
bug or any others) with another process. Someone has to really sit down and cleanup manually.
This feature should pave the way for us to have a single process entity corresponding to a
processing need and allow for natural evolution with system tracking and maintaining history
of those changes. The status quo is quite broken IMHO. 

>> a lot of features developed in falcon took the immutability of entity definition
for past instances as given

I think this assumption isn't valid. Past instances are mutable. People do rerun older instances,
even worse they create newer process entities to re-run old instances.

> Effective time in Entity updates.
> ---------------------------------
>                 Key: FALCON-1406
>                 URL: https://issues.apache.org/jira/browse/FALCON-1406
>             Project: Falcon
>          Issue Type: New Feature
>            Reporter: sandeep samudrala
>            Assignee: sandeep samudrala
>         Attachments: FALCON-1406-initial.patch, effective_time_in_entity_updates.pdf
> Effective time with entity updates needs to be provided even with past time too. There
was effective time capability provided in the past which gives the functionality to set an
effective time for an entity with only current or future time(now + delay), which could not
solve all the issues. 
> Following are few scenarios which would require effective time to be available with time
back in past.
> a) New code being deployed for an incompatible input data set which would leave instances
with old code and new data.
> b) Bad code being pushed for which, the entity should be able to go back in time to replay(rerun)
with new code.
> c) Orchestration level changes(good/bad) would need functionality to go back in time
to start with.
> For reference: Linking all the Jiras that have been worked upon around effective time
> https://issues.apache.org/jira/browse/FALCON-374
> https://issues.apache.org/jira/browse/FALCON-297

This message was sent by Atlassian JIRA

View raw message