Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAN4cK6qeJWT5z9bE7zPnAL45yKY6gh_OoSCHBVvHFRWLpNj=mw@mail.gmail.com>
References: 
 <CAKt=akE0e7cXtV4oWK9JVP==huDqhhMAqSZ0PdNzUcE0+HTaXw@mail.gmail.com>
 <CA+QE8bed52TUbPOEqd=vKYrBxwTvT=cqR-gTE8CrkASxO1X4PA@mail.gmail.com>
 <CAKaBFB=BDpJ_8e-3ow8uS=CppC4NU-UG1tBTxzpEuR5+Rm_JPQ@mail.gmail.com>
 <CAKt=akGU-Udkf_CnCt54-3WeZTgvcdbeDQUZQHOQyyWY3KXjsg@mail.gmail.com>
 <CAA7LveGStqMz5+SxK+0aKhFbCKx2s6A9id3h+h4aHwk9HAqO_g@mail.gmail.com>
 <CAKaBFBkSV-oxjtUh4TH1CEj6Xdxqo=2PEcYDK=Hngmeiuu6M4A@mail.gmail.com>
 <CAA7LveFHBevJ=Zn3YohY5HvKLHuaYg0Sdx8hK9KaysUZ5H8PzA@mail.gmail.com>
 <CAKt=akFcRLUZP6Xyz-BX-oB_80MERfBL=Nkt4Y8AxcUEbm1UNg@mail.gmail.com>
 <CAA7LveGnFknw-rx4ZZR0fL3xoDcrPC77oCzuwQSeKVC9z_WRjA@mail.gmail.com>
 <CAN4cK6qeJWT5z9bE7zPnAL45yKY6gh_OoSCHBVvHFRWLpNj=mw@mail.gmail.com>
From: Pramod Immaneni <pramod@datatorrent.com>
Date: Wed, 23 Sep 2015 00:02:52 -0700
Message-ID: 
 <CAA7LveHri129Kz3sB8jh=64bzuSBFR2oY+2q5xT2x+aVMcpJEg@mail.gmail.com>
Subject: Re: Assign property to partition instances of Operator
To: dev@apex.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a11440fb8be8cd4052064b54f

--001a11440fb8be8cd4052064b54f
Content-Type: text/plain; charset=UTF-8

Yes two ways separate operator or repartition.

On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <gaurav@datatorrent.com>
wrote:

> Pramod ,
>
> That is what I was thinking that Operator requests can also be received in
> different windows...Looks like adding another operator is safe solution
>
> Thanks
> -Gaurav
>
> On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
> > The operator request would also be received in different windows.
> >
> > On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
> > priyanka@datatorrent.com>
> > wrote:
> >
> > > Promod,
> > >
> > > You are right about applying the value in same window. And yes I don't
> > want
> > > to change number of partitions or even redeploy any partition for this.
> > > I need to explore on Chandini's suggestion of using StatsListener.
> > >
> > > -Priyanka
> > >
> > > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > > > I am not sure about priyanka's requirements but typically you would
> > want
> > > to
> > > > apply the value in the same window for all partitions for consistency
> > > with
> > > > the calculations. This could be done two ways using a separate
> logical
> > > > operator feeding the value to all partitions so that the value
> appears
> > in
> > > > the same window or repartitioning the existing set and setting the
> > value
> > > > during repartitioning. In the latter case you are just repartitioning
> > so
> > > > that the partitions start from same point with the new value and not
> to
> > > > change the number of partitions.
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
> > chandni@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > Pramod,
> > > > >
> > > > > Tim's discussion was about setting a property on multiple operators
> > > > > (different logical operators).
> > > > > Priyanka's question is about different partitions of an operator
> > > > >
> > > > > Let's say they are 5 partitions of an operator and we need to
> change
> > an
> > > > int
> > > > > property.  There are following ways:
> > > > >
> > > > > 1. Divide the int property by 5 and issue set-physical operator
> > > property
> > > > > command for each partition. This is manual.
> > > > >
> > > > > 2. Set the logical operator property but wait till partition
> happens.
> > > > This
> > > > > will require partitioning to be triggered for which a StatsListener
> > > needs
> > > > > to be implemented. So if partitioning never happens this will never
> > be
> > > > > triggered. Also un-deploying and deploying a partition just to
> > change a
> > > > > property on the physical partition seems un-necessary
> > > > >
> > > > > 3. Implementing StatsListener to send operator request and not
> > > > re-partition
> > > > > seems to be appropriate way.
> > > > >
> > > > > Chandni
> > > > >
> > > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> > > > pramod@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Priyanka you want to divide the value among existing partitions
> > right
> > > > and
> > > > > > not necessarily change the number of partitions. To this end
> Ashwin
> > > is
> > > > > > suggesting makes sense. However different partitions can be at
> > > > different
> > > > > > windows and will see the value change in different windows. You
> > need
> > > a
> > > > > > separate operator to receive this change and propagate it to the
> > > > > partitions
> > > > > > via data flow. Tim discussed this over the last weekend in a
> series
> > > of
> > > > > > emails.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > > > > > priyanka@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for suggestions.
> > > > > > >
> > > > > > > Ashwin, The option you proposed is doable, but then each
> operator
> > > > which
> > > > > > > needs to address such use case will have to track the number of
> > > > > > partitions.
> > > > > > > So we were looking for some common mechanism.
> > > > > > > Chandni, Let me try the res.operatorRequests, that could solve
> > the
> > > > > > problem.
> > > > > > >
> > > > > > > -Priyanka
> > > > > > >
> > > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > > > > chandni@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Priyanka,
> > > > > > > >
> > > > > > > > As Isha mentioned you will can implement a StatsListener.
> > However
> > > > > > instead
> > > > > > > > of setting res.repartitionRequired = true, you can add a
> > > > > > > > StatsListener.OperatorRequest to res.operatorsRequests. This
> > will
> > > > not
> > > > > > > > re-deploy the instances.
> > > > > > > >
> > > > > > > > The OperatorRequest that you will need to implement will have
> > to
> > > > > change
> > > > > > > the
> > > > > > > > property value.
> > > > > > > >
> > > > > > > > You can refer to AutoMetricTest for an example.
> > > > > > > >
> > > > > > > > - Chandni
> > > > > > > >
> > > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > > > > > > > ashwinchandrap@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Why not maintain a variable containing the number of
> > partitions
> > > > in
> > > > > > each
> > > > > > > > > partition; and when you set the property, just divide it by
> > > > number
> > > > > of
> > > > > > > > > partitions variable? Does it not solve the problem?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Ashwin.
> > > > > > > > >
> > > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > > > > > > priyanka@datatorrent.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I have an integer property named "intProp". This property
> > is
> > > > > > > applicable
> > > > > > > > > to
> > > > > > > > > > an operator A. I can set the property from dtcli using
> > > > > > > > > > "set-operator-property operator-name property-name
> > > > > property-value"
> > > > > > > and
> > > > > > > > > this
> > > > > > > > > > will get applied to all instances of opeartor A.
> > > > > > > > > >
> > > > > > > > > > Now, I want to apply this property in a way that it's
> > equally
> > > > > > divided
> > > > > > > > > > amongst all instances of operator A. For example,  if
> > > > intProp=100
> > > > > > and
> > > > > > > > > > number of partitions=4 then intProp=25 should be set on
> > each
> > > > > > instance
> > > > > > > > of
> > > > > > > > > > operator A.
> > > > > > > > > >
> > > > > > > > > > Is there a way to achieve this?
> > > > > > > > > >
> > > > > > > > > > I have tried following approach to do this: I have
> > overriden
> > > > > > > > > > definePartitions of Operator, once we do partitioning I
> am
> > > > > setting
> > > > > > > > right
> > > > > > > > > > value on each operator instance based on new partition
> > count.
> > > > The
> > > > > > > > > > limitation here is that the value is applied during
> > > > > > definePartitions.
> > > > > > > > If
> > > > > > > > > > the value is updated dynamically it won't be applied till
> > > > > > > > > definePartitions
> > > > > > > > > > is called again.
> > > > > > > > > >
> > > > > > > > > > -Priyanka
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Ashwin.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--001a11440fb8be8cd4052064b54f--