Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
From: Gaurav Gupta <gaurav@datatorrent.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_E7F7E908-4F37-42E6-AB75-E22E27D5055A"
Message-Id: <0776560B-0B86-44A5-A8AA-8BB154095AE0@datatorrent.com>
Mime-Version: 1.0 (Mac OS X Mail 9.0 \(3094\))
Subject: Re: dynamic application properties proposal
Date: Thu, 1 Oct 2015 11:13:43 -0700
References: 
 <CAA7LveH4DL6imgHTq9Jb_cSy+4LTq-xvas-rnTKq18LjgpDf6A@mail.gmail.com>
 <CA+QE8bf=UkdKeDipOugXq0bBO+hd7Vhm8bgoi2FvPnocgoPn2A@mail.gmail.com>
 <CAA7LveHBUSXY3aWaJVSFOosDSBT=r_pBh9a0nDKtreqm+u61eA@mail.gmail.com>
To: dev@apex.incubator.apache.org
In-Reply-To: 
 <CAA7LveHBUSXY3aWaJVSFOosDSBT=r_pBh9a0nDKtreqm+u61eA@mail.gmail.com>

--Apple-Mail=_E7F7E908-4F37-42E6-AB75-E22E27D5055A
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Pramod,

The new special property change tuple will be send to all the Operators =
and all the operators will have to check if the property change is =
applicable for it. Although such requests may be very few, but is there =
a way to optimize it?

Thanks
- Gaurav

> On Sep 28, 2015, at 3:44 PM, Pramod Immaneni <pramod@datatorrent.com> =
wrote:
>=20
> At the platform level that cannot be guaranteed as your operator =
controls
> and manages reading of the data. However it is not difficult to =
envision
> writing an operator that would pick up a new dataset when property is
> changed.
>=20
> On Mon, Sep 28, 2015 at 3:33 PM, Ashwin Chandra Putta <
> ashwinchandrap@gmail.com> wrote:
>=20
>> Great, looking forward to these changes. Does it also provide a =
guarantee
>> on which properties are used for which input data sets?
>>=20
>> Few use case examples:
>> - set property between reads of different batches of files. Say, =
applying
>> batch name property before processing the next batch of files.
>> - load new configuration file for csv parser before processing next =
set of
>> data.
>> - apply new regex before parsing next stream of tuples.
>> etc.
>>=20
>> One approach to allow this is to emit subsequent tuples only starting =
next
>> window after the window in which property change is made. That way, =
the
>> boundaries between data sets is fixed and property change is done in
>> between. The user will now have a guarantee on which property value =
is used
>> on any given tuple.
>>=20
>> Thoughts?
>>=20
>> Regards,
>> Ashwin.
>>=20
>> On Mon, Sep 28, 2015 at 10:17 AM, Pramod Immaneni =
<pramod@datatorrent.com>
>> wrote:
>>=20
>>> Apex support modification of operator properties at runtime but the
>> current
>>> implemenations has the following shortcomings.
>>>=20
>>> 1. Property is not set across all partitions on the same window as
>>> individual partitions can be on different windows when property =
change is
>>> initiated from client resulting in inconsistency of data for those
>> windows.
>>> I am being generous using the word inconsistent.
>>> 2. Sometimes properties need to be set on more than one logical =
operators
>>> at the same time to achieve the change the user is seeking. Today =
they
>> will
>>> be two separate changes happening on two different windows again
>> resulting
>>> in inconsistent data for some windows. These would need to happen as =
a
>>> single transaction.
>>> 3. If there is an operator failure before a committed checkpoint =
after an
>>> operator property is dynamically changed the operator will restart =
with
>> the
>>> old property and the change will not be re-applied.
>>>=20
>>> Tim and myself did some brainstorming and we have a proposal to =
overcome
>>> these shortcomings. The main problem in all the above cases is that =
the
>>> property changes are happening out-of-band of data flow and hence
>>> independent of windowing. The proposal is to bring the property =
change
>>> request into the in-band dataflow so that they are handled =
consistently
>>> with windowing and handled distributively.
>>>=20
>>> The idea is to inject a special property change tuple containing the
>>> property changes and the identification information of the =
operator's
>> they
>>> affect into the dataflow at the input operator. The tuple will be
>> injected
>>> at window boundary after end window and before begin window and as =
this
>>> tuple flows through the DAG the intended operators properties will =
be
>>> modifed. They will all be modified consistently at the same window. =
The
>>> tuple can contain more than one property changes for more than one
>> logical
>>> operators and the change will be applied consistently to the =
different
>>> logical operators at the same window. In case of failure the replay =
of
>>> tuples will ensure that the property change gets reapplied at the =
correct
>>> window.
>>>=20
>>> Please give your feedback and input on what you think about this
>> proposal.
>>>=20
>>> Thanks
>>>=20
>>=20
>>=20
>>=20
>> --
>>=20
>> Regards,
>> Ashwin.
>>=20


--Apple-Mail=_E7F7E908-4F37-42E6-AB75-E22E27D5055A--