apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Gupta <gau...@datatorrent.com>
Subject Re: Assign property to partition instances of Operator
Date Wed, 23 Sep 2015 07:03:42 GMT
Pramod,

What happens if the Separate Operator that is used to propagate this change
is on say window 1000 and all the instances of Operator A are at 300..At
this time user changes value of property. Then the new value will be
delayed to Operator A instances by 700 windows

Thanks
-Gaurav

On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <gaurav@datatorrent.com>
wrote:

> Pramod ,
>
> That is what I was thinking that Operator requests can also be received in
> different windows...Looks like adding another operator is safe solution
>
> Thanks
> -Gaurav
>
> On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
>> The operator request would also be received in different windows.
>>
>> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
>> priyanka@datatorrent.com>
>> wrote:
>>
>> > Promod,
>> >
>> > You are right about applying the value in same window. And yes I don't
>> want
>> > to change number of partitions or even redeploy any partition for this.
>> > I need to explore on Chandini's suggestion of using StatsListener.
>> >
>> > -Priyanka
>> >
>> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
>> pramod@datatorrent.com>
>> > wrote:
>> >
>> > > I am not sure about priyanka's requirements but typically you would
>> want
>> > to
>> > > apply the value in the same window for all partitions for consistency
>> > with
>> > > the calculations. This could be done two ways using a separate logical
>> > > operator feeding the value to all partitions so that the value
>> appears in
>> > > the same window or repartitioning the existing set and setting the
>> value
>> > > during repartitioning. In the latter case you are just repartitioning
>> so
>> > > that the partitions start from same point with the new value and not
>> to
>> > > change the number of partitions.
>> > >
>> > > Thanks
>> > >
>> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
>> chandni@datatorrent.com
>> > >
>> > > wrote:
>> > >
>> > > > Pramod,
>> > > >
>> > > > Tim's discussion was about setting a property on multiple operators
>> > > > (different logical operators).
>> > > > Priyanka's question is about different partitions of an operator
>> > > >
>> > > > Let's say they are 5 partitions of an operator and we need to
>> change an
>> > > int
>> > > > property.  There are following ways:
>> > > >
>> > > > 1. Divide the int property by 5 and issue set-physical operator
>> > property
>> > > > command for each partition. This is manual.
>> > > >
>> > > > 2. Set the logical operator property but wait till partition
>> happens.
>> > > This
>> > > > will require partitioning to be triggered for which a StatsListener
>> > needs
>> > > > to be implemented. So if partitioning never happens this will never
>> be
>> > > > triggered. Also un-deploying and deploying a partition just to
>> change a
>> > > > property on the physical partition seems un-necessary
>> > > >
>> > > > 3. Implementing StatsListener to send operator request and not
>> > > re-partition
>> > > > seems to be appropriate way.
>> > > >
>> > > > Chandni
>> > > >
>> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
>> > > pramod@datatorrent.com>
>> > > > wrote:
>> > > >
>> > > > > Priyanka you want to divide the value among existing partitions
>> right
>> > > and
>> > > > > not necessarily change the number of partitions. To this end
>> Ashwin
>> > is
>> > > > > suggesting makes sense. However different partitions can be at
>> > > different
>> > > > > windows and will see the value change in different windows. You
>> need
>> > a
>> > > > > separate operator to receive this change and propagate it to
the
>> > > > partitions
>> > > > > via data flow. Tim discussed this over the last weekend in a
>> series
>> > of
>> > > > > emails.
>> > > > >
>> > > > > Thanks
>> > > > >
>> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
>> > > > > priyanka@datatorrent.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Thanks for suggestions.
>> > > > > >
>> > > > > > Ashwin, The option you proposed is doable, but then each
>> operator
>> > > which
>> > > > > > needs to address such use case will have to track the number
of
>> > > > > partitions.
>> > > > > > So we were looking for some common mechanism.
>> > > > > > Chandni, Let me try the res.operatorRequests, that could
solve
>> the
>> > > > > problem.
>> > > > > >
>> > > > > > -Priyanka
>> > > > > >
>> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
>> > > > chandni@datatorrent.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Priyanka,
>> > > > > > >
>> > > > > > > As Isha mentioned you will can implement a StatsListener.
>> However
>> > > > > instead
>> > > > > > > of setting res.repartitionRequired = true, you can
add a
>> > > > > > > StatsListener.OperatorRequest to res.operatorsRequests.
This
>> will
>> > > not
>> > > > > > > re-deploy the instances.
>> > > > > > >
>> > > > > > > The OperatorRequest that you will need to implement
will have
>> to
>> > > > change
>> > > > > > the
>> > > > > > > property value.
>> > > > > > >
>> > > > > > > You can refer to AutoMetricTest for an example.
>> > > > > > >
>> > > > > > > - Chandni
>> > > > > > >
>> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta
<
>> > > > > > > ashwinchandrap@gmail.com> wrote:
>> > > > > > >
>> > > > > > > > Why not maintain a variable containing the number
of
>> partitions
>> > > in
>> > > > > each
>> > > > > > > > partition; and when you set the property, just
divide it by
>> > > number
>> > > > of
>> > > > > > > > partitions variable? Does it not solve the problem?
>> > > > > > > >
>> > > > > > > > Regards,
>> > > > > > > > Ashwin.
>> > > > > > > >
>> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale
<
>> > > > > > > priyanka@datatorrent.com
>> > > > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I have an integer property named "intProp".
This property
>> is
>> > > > > > applicable
>> > > > > > > > to
>> > > > > > > > > an operator A. I can set the property from
dtcli using
>> > > > > > > > > "set-operator-property operator-name property-name
>> > > > property-value"
>> > > > > > and
>> > > > > > > > this
>> > > > > > > > > will get applied to all instances of opeartor
A.
>> > > > > > > > >
>> > > > > > > > > Now, I want to apply this property in a way
that it's
>> equally
>> > > > > divided
>> > > > > > > > > amongst all instances of operator A. For
example,  if
>> > > intProp=100
>> > > > > and
>> > > > > > > > > number of partitions=4 then intProp=25 should
be set on
>> each
>> > > > > instance
>> > > > > > > of
>> > > > > > > > > operator A.
>> > > > > > > > >
>> > > > > > > > > Is there a way to achieve this?
>> > > > > > > > >
>> > > > > > > > > I have tried following approach to do this:
I have
>> overriden
>> > > > > > > > > definePartitions of Operator, once we do
partitioning I am
>> > > > setting
>> > > > > > > right
>> > > > > > > > > value on each operator instance based on
new partition
>> count.
>> > > The
>> > > > > > > > > limitation here is that the value is applied
during
>> > > > > definePartitions.
>> > > > > > > If
>> > > > > > > > > the value is updated dynamically it won't
be applied till
>> > > > > > > > definePartitions
>> > > > > > > > > is called again.
>> > > > > > > > >
>> > > > > > > > > -Priyanka
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > >
>> > > > > > > > Regards,
>> > > > > > > > Ashwin.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message