apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sasha Parfenov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (APEXCORE-163) Dynamic application property changes
Date Tue, 29 Dec 2015 23:52:49 GMT

     [ https://issues.apache.org/jira/browse/APEXCORE-163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sasha Parfenov updated APEXCORE-163:
------------------------------------
    Labels: roadmap  (was: )

> Dynamic application property changes
> ------------------------------------
>
>                 Key: APEXCORE-163
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-163
>             Project: Apache Apex Core
>          Issue Type: Improvement
>            Reporter: Pramod Immaneni
>            Assignee: Pramod Immaneni
>              Labels: roadmap
>
> Apex support modification of operator properties at runtime but the current implemenations
has the following shortcomings.
> 1. Property is not set across all partitions on the same window as individual partitions
can be on different windows when property change is initiated from client resulting in inconsistency
of data for those windows. I am being generous using the word inconsistent.
> 2. Sometimes properties need to be set on more than one logical operators at the same
time to achieve the change the user is seeking. Today they will be two separate changes happening
on two different windows again resulting in inconsistent data for some windows. These would
need to happen as a single transaction.
> 3. If there is an operator failure before a committed checkpoint after an operator property
is dynamically changed the operator will restart with the old property and the change will
not be re-applied.
> Tim and myself did some brainstorming and we have a proposal to overcome these shortcomings.
The main problem in all the above cases is that the property changes are happening out-of-band
of data flow and hence independent of windowing. The proposal is to bring the property change
request into the in-band dataflow so that they are handled consistently with windowing and
handled distributively.
> The idea is to inject a special property change tuple containing the property changes
and the identification information of the operator's they affect into the dataflow at the
input operator. The tuple will be injected at window boundary after end window and before
begin window and as this tuple flows through the DAG the intended operators properties will
be modifed. They will all be modified consistently at the same window. The tuple can contain
more than one property changes for more than one logical operators and the change will be
applied consistently to the different logical operators at the same window. In case of failure
the replay of tuples will ensure that the property change gets reapplied at the correct window.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message