apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: Assign property to partition instances of Operator
Date Wed, 23 Sep 2015 07:02:52 GMT
Yes two ways separate operator or repartition.

On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <gaurav@datatorrent.com>
wrote:

> Pramod ,
>
> That is what I was thinking that Operator requests can also be received in
> different windows...Looks like adding another operator is safe solution
>
> Thanks
> -Gaurav
>
> On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
> > The operator request would also be received in different windows.
> >
> > On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
> > priyanka@datatorrent.com>
> > wrote:
> >
> > > Promod,
> > >
> > > You are right about applying the value in same window. And yes I don't
> > want
> > > to change number of partitions or even redeploy any partition for this.
> > > I need to explore on Chandini's suggestion of using StatsListener.
> > >
> > > -Priyanka
> > >
> > > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > > > I am not sure about priyanka's requirements but typically you would
> > want
> > > to
> > > > apply the value in the same window for all partitions for consistency
> > > with
> > > > the calculations. This could be done two ways using a separate
> logical
> > > > operator feeding the value to all partitions so that the value
> appears
> > in
> > > > the same window or repartitioning the existing set and setting the
> > value
> > > > during repartitioning. In the latter case you are just repartitioning
> > so
> > > > that the partitions start from same point with the new value and not
> to
> > > > change the number of partitions.
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
> > chandni@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > Pramod,
> > > > >
> > > > > Tim's discussion was about setting a property on multiple operators
> > > > > (different logical operators).
> > > > > Priyanka's question is about different partitions of an operator
> > > > >
> > > > > Let's say they are 5 partitions of an operator and we need to
> change
> > an
> > > > int
> > > > > property.  There are following ways:
> > > > >
> > > > > 1. Divide the int property by 5 and issue set-physical operator
> > > property
> > > > > command for each partition. This is manual.
> > > > >
> > > > > 2. Set the logical operator property but wait till partition
> happens.
> > > > This
> > > > > will require partitioning to be triggered for which a StatsListener
> > > needs
> > > > > to be implemented. So if partitioning never happens this will never
> > be
> > > > > triggered. Also un-deploying and deploying a partition just to
> > change a
> > > > > property on the physical partition seems un-necessary
> > > > >
> > > > > 3. Implementing StatsListener to send operator request and not
> > > > re-partition
> > > > > seems to be appropriate way.
> > > > >
> > > > > Chandni
> > > > >
> > > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> > > > pramod@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Priyanka you want to divide the value among existing partitions
> > right
> > > > and
> > > > > > not necessarily change the number of partitions. To this end
> Ashwin
> > > is
> > > > > > suggesting makes sense. However different partitions can be
at
> > > > different
> > > > > > windows and will see the value change in different windows.
You
> > need
> > > a
> > > > > > separate operator to receive this change and propagate it to
the
> > > > > partitions
> > > > > > via data flow. Tim discussed this over the last weekend in a
> series
> > > of
> > > > > > emails.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > > > > > priyanka@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for suggestions.
> > > > > > >
> > > > > > > Ashwin, The option you proposed is doable, but then each
> operator
> > > > which
> > > > > > > needs to address such use case will have to track the number
of
> > > > > > partitions.
> > > > > > > So we were looking for some common mechanism.
> > > > > > > Chandni, Let me try the res.operatorRequests, that could
solve
> > the
> > > > > > problem.
> > > > > > >
> > > > > > > -Priyanka
> > > > > > >
> > > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > > > > chandni@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Priyanka,
> > > > > > > >
> > > > > > > > As Isha mentioned you will can implement a StatsListener.
> > However
> > > > > > instead
> > > > > > > > of setting res.repartitionRequired = true, you can
add a
> > > > > > > > StatsListener.OperatorRequest to res.operatorsRequests.
This
> > will
> > > > not
> > > > > > > > re-deploy the instances.
> > > > > > > >
> > > > > > > > The OperatorRequest that you will need to implement
will have
> > to
> > > > > change
> > > > > > > the
> > > > > > > > property value.
> > > > > > > >
> > > > > > > > You can refer to AutoMetricTest for an example.
> > > > > > > >
> > > > > > > > - Chandni
> > > > > > > >
> > > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta
<
> > > > > > > > ashwinchandrap@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Why not maintain a variable containing the number
of
> > partitions
> > > > in
> > > > > > each
> > > > > > > > > partition; and when you set the property, just
divide it by
> > > > number
> > > > > of
> > > > > > > > > partitions variable? Does it not solve the problem?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Ashwin.
> > > > > > > > >
> > > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale
<
> > > > > > > > priyanka@datatorrent.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I have an integer property named "intProp".
This property
> > is
> > > > > > > applicable
> > > > > > > > > to
> > > > > > > > > > an operator A. I can set the property from
dtcli using
> > > > > > > > > > "set-operator-property operator-name property-name
> > > > > property-value"
> > > > > > > and
> > > > > > > > > this
> > > > > > > > > > will get applied to all instances of opeartor
A.
> > > > > > > > > >
> > > > > > > > > > Now, I want to apply this property in a
way that it's
> > equally
> > > > > > divided
> > > > > > > > > > amongst all instances of operator A. For
example,  if
> > > > intProp=100
> > > > > > and
> > > > > > > > > > number of partitions=4 then intProp=25 should
be set on
> > each
> > > > > > instance
> > > > > > > > of
> > > > > > > > > > operator A.
> > > > > > > > > >
> > > > > > > > > > Is there a way to achieve this?
> > > > > > > > > >
> > > > > > > > > > I have tried following approach to do this:
I have
> > overriden
> > > > > > > > > > definePartitions of Operator, once we do
partitioning I
> am
> > > > > setting
> > > > > > > > right
> > > > > > > > > > value on each operator instance based on
new partition
> > count.
> > > > The
> > > > > > > > > > limitation here is that the value is applied
during
> > > > > > definePartitions.
> > > > > > > > If
> > > > > > > > > > the value is updated dynamically it won't
be applied till
> > > > > > > > > definePartitions
> > > > > > > > > > is called again.
> > > > > > > > > >
> > > > > > > > > > -Priyanka
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Ashwin.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message