apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject dynamic application properties proposal
Date Mon, 28 Sep 2015 17:17:03 GMT
Apex support modification of operator properties at runtime but the current
implemenations has the following shortcomings.

1. Property is not set across all partitions on the same window as
individual partitions can be on different windows when property change is
initiated from client resulting in inconsistency of data for those windows.
I am being generous using the word inconsistent.
2. Sometimes properties need to be set on more than one logical operators
at the same time to achieve the change the user is seeking. Today they will
be two separate changes happening on two different windows again resulting
in inconsistent data for some windows. These would need to happen as a
single transaction.
3. If there is an operator failure before a committed checkpoint after an
operator property is dynamically changed the operator will restart with the
old property and the change will not be re-applied.

Tim and myself did some brainstorming and we have a proposal to overcome
these shortcomings. The main problem in all the above cases is that the
property changes are happening out-of-band of data flow and hence
independent of windowing. The proposal is to bring the property change
request into the in-band dataflow so that they are handled consistently
with windowing and handled distributively.

The idea is to inject a special property change tuple containing the
property changes and the identification information of the operator's they
affect into the dataflow at the input operator. The tuple will be injected
at window boundary after end window and before begin window and as this
tuple flows through the DAG the intended operators properties will be
modifed. They will all be modified consistently at the same window. The
tuple can contain more than one property changes for more than one logical
operators and the change will be applied consistently to the different
logical operators at the same window. In case of failure the replay of
tuples will ensure that the property change gets reapplied at the correct
window.

Please give your feedback and input on what you think about this proposal.

Thanks

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message