falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1406) Effective time in Entity updates.
Date Wed, 23 Nov 2016 18:32:58 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15690980#comment-15690980
] 

Ajay Yadava commented on FALCON-1406:
-------------------------------------

Hi  [~sandeep.samudrala],
In the interest of saving time, I and Srikanth had a detailed offline discussion on why versioning
is required. I hope it will be much easier for you to sync up with him and understand the
scenarios in detail. It will be nice to capture all those scenarios in the design doc e.g.

1. How does it impact other entities? Why this feature does or doesn't make sense for them?
2. How will the following systems / features work after this change: -
graph DB, relational databases maintaining instance states, SLA (feed and process both), backlog,
alerting, monitoring, instance dependency, triage, instance listing, entity lineage, recipes,
late re-run, and retries.

Just to clarify, I am not implying any changes to these systems, just want the design doc
to capture your thoughts and discussion on these aspects, and we can make progress from there.

More work coming your way :P On a serious note, really appreciate your effort and thanks for
taking it up. It is a huge ask and has been a heroic effort to bring it up to this level.


> Effective time in Entity updates.
> ---------------------------------
>
>                 Key: FALCON-1406
>                 URL: https://issues.apache.org/jira/browse/FALCON-1406
>             Project: Falcon
>          Issue Type: New Feature
>            Reporter: sandeep samudrala
>            Assignee: sandeep samudrala
>         Attachments: FALCON-1406-initial.patch, effective_time_in_entity_updates.pdf
>
>
> Effective time with entity updates needs to be provided even with past time too. There
was effective time capability provided in the past which gives the functionality to set an
effective time for an entity with only current or future time(now + delay), which could not
solve all the issues. 
> Following are few scenarios which would require effective time to be available with time
back in past.
> a) New code being deployed for an incompatible input data set which would leave instances
with old code and new data.
> b) Bad code being pushed for which, the entity should be able to go back in time to replay(rerun)
with new code.
> c) Orchestration level changes(good/bad) would need functionality to go back in time
to start with.
> For reference: Linking all the Jiras that have been worked upon around effective time
.
> https://issues.apache.org/jira/browse/FALCON-374
> https://issues.apache.org/jira/browse/FALCON-297



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message