apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: Assign property to partition instances of Operator
Date Wed, 23 Sep 2015 06:02:00 GMT
Priyanka you want to divide the value among existing partitions right and
not necessarily change the number of partitions. To this end Ashwin is
suggesting makes sense. However different partitions can be at different
windows and will see the value change in different windows. You need a
separate operator to receive this change and propagate it to the partitions
via data flow. Tim discussed this over the last weekend in a series of
emails.

Thanks

On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <priyanka@datatorrent.com>
wrote:

> Thanks for suggestions.
>
> Ashwin, The option you proposed is doable, but then each operator which
> needs to address such use case will have to track the number of partitions.
> So we were looking for some common mechanism.
> Chandni, Let me try the res.operatorRequests, that could solve the problem.
>
> -Priyanka
>
> On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <chandni@datatorrent.com>
> wrote:
>
> > Hi Priyanka,
> >
> > As Isha mentioned you will can implement a StatsListener. However instead
> > of setting res.repartitionRequired = true, you can add a
> > StatsListener.OperatorRequest to res.operatorsRequests. This will not
> > re-deploy the instances.
> >
> > The OperatorRequest that you will need to implement will have to change
> the
> > property value.
> >
> > You can refer to AutoMetricTest for an example.
> >
> > - Chandni
> >
> > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > ashwinchandrap@gmail.com> wrote:
> >
> > > Why not maintain a variable containing the number of partitions in each
> > > partition; and when you set the property, just divide it by number of
> > > partitions variable? Does it not solve the problem?
> > >
> > > Regards,
> > > Ashwin.
> > >
> > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > priyanka@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > I have an integer property named "intProp". This property is
> applicable
> > > to
> > > > an operator A. I can set the property from dtcli using
> > > > "set-operator-property operator-name property-name property-value"
> and
> > > this
> > > > will get applied to all instances of opeartor A.
> > > >
> > > > Now, I want to apply this property in a way that it's equally divided
> > > > amongst all instances of operator A. For example,  if intProp=100 and
> > > > number of partitions=4 then intProp=25 should be set on each instance
> > of
> > > > operator A.
> > > >
> > > > Is there a way to achieve this?
> > > >
> > > > I have tried following approach to do this: I have overriden
> > > > definePartitions of Operator, once we do partitioning I am setting
> > right
> > > > value on each operator instance based on new partition count. The
> > > > limitation here is that the value is applied during definePartitions.
> > If
> > > > the value is updated dynamically it won't be applied till
> > > definePartitions
> > > > is called again.
> > > >
> > > > -Priyanka
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashwin.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message