Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAERdzhYbsx6LG-j0FwwiL0tTQfWKuuE5FAUj++o3DEHKmWtd0g@mail.gmail.com>
References: 
 <CAMqituPxN1f0ZQ9h5PmRg+SBKSa2Wdk3KQ8UY5O7NPRnw5Jvcw@mail.gmail.com>
	<CAERdzhZR4zSPYYGMiuKD5Pw5UVDeVRrpSNt4==-cYBtkju4nYw@mail.gmail.com>
	<CAMqituNZxYsEYZg_m0gZv4gSJhzFrXrtwmqOJS+9Ec=wU3wsaQ@mail.gmail.com>
	<CAJ484BikyG-kRA6TtkzxZYWoHKQdO8K6hsjjx89ZG5OQ_rvtOQ@mail.gmail.com>
	<CAMqituNFyWvhQQoxvDbv++2dLZ4L3P5mN9-GeSR=2eux9nGVOQ@mail.gmail.com>
	<CAERdzhYbsx6LG-j0FwwiL0tTQfWKuuE5FAUj++o3DEHKmWtd0g@mail.gmail.com>
Date: Thu, 17 Sep 2015 15:45:32 -0700
Message-ID: 
 <CAKaBFBkdJ7ZthHZuAaSO6z9eK0B-Y_Y1VdAMOthYCKsiC5ronA@mail.gmail.com>
Subject: Re: Supporting iterations in Apex
From: Chandni Singh <chandni@datatorrent.com>
To: dev@apex.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a11c25e3c335d57051ff92c19

--001a11c25e3c335d57051ff92c19
Content-Type: text/plain; charset=UTF-8

+1 for ITERATION_WINDOW_OFFSET.  We are trying to enable APEX for Iterative
Machine Learning and this sounds clearer/less confusing to me.

On Thu, Sep 17, 2015 at 3:39 PM, Chetan Narsude <chetan@datatorrent.com>
wrote:

> Iteration implies that something is looping over. Whereas that's just one
> use case of this functionality. One can take the output of an upstream
> operator and give it to input of the downstream operator.
>
> AFAIK, DELAY  is very well understood concept in event processing and
> analogous to how we intend to use it.
>
>
> On Wed, Sep 16, 2015 at 5:32 PM, David Yan <david@datatorrent.com> wrote:
>
> > I think keeping the word ITERATION is clearer to the users because that's
> > what it is for.
> > The user wouldn't think he/she is trying to "delay" something...
> > In any case, I am fine either way :)
> >
> > David
> >
> > On Wed, Sep 16, 2015 at 5:12 PM, Munagala Ramanath <ram@datatorrent.com>
> > wrote:
> >
> > > I like ITERATION_WINDOW_OFFSET.
> > >
> > > Ram
> > >
> > > On Wed, Sep 16, 2015 at 4:42 PM, David Yan <david@datatorrent.com>
> > wrote:
> > >
> > > > Thanks Chetan.
> > > >
> > > > Can you point me to the location of Deduper code that may be helpful
> > with
> > > > the recovery implementation?
> > > >
> > > > Does anyone have any opinion on the renaming of
> ITERATION_WINDOW_COUNT?
> > > > DELAY_BY_WINDOW_COUNT? DELAY_WINDOW_COUNT?
> > > >
> > > > David
> > > >
> > > > On Wed, Sep 16, 2015 at 2:21 PM, Chetan Narsude <
> > chetan@datatorrent.com>
> > > > wrote:
> > > >
> > > > > David,
> > > > >
> > > > >  I have 3 comments:
> > > > >
> > > > > 1. The "ahead window" phrase you discussed above is really behind
> > > window.
> > > > > With Apex, the windows which are ahead are the windows with smaller
> > > > window
> > > > > Id. smaller window ids are followed by bigger window ids.
> > > > >
> > > > > 2.  ITERATION_WINDOW_COUNT sounds like a misnomer. IMO, It  should
> be
> > > > > something akin to DELAY_BY_WINDOW_COUNT as you are delaying the
> > events
> > > by
> > > > > those many windows. You are not iterating over them as many times.
> It
> > > > also
> > > > > resonates with PortContext.SLIDE_BY_WINDOW_COUNT
> > > > >
> > > > > 3. Deduper has similar requirement where large amount of data
> > > > (potentially
> > > > > even larger) needs to be partitioned. You can borrow the idea/code
> > from
> > > > > there. And perhaps abstract the code to be reusable.
> > > > >
> > > > > HTH.
> > > > >
> > > > > --
> > > > > Chetan
> > > > >
> > > > > On Wed, Sep 16, 2015 at 1:44 PM, David Yan <david@datatorrent.com>
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > One current disadvantage of Apex is the inability to do
> iterations
> > > and
> > > > > > machine learning algorithms because we don't allow loops in the
> > > > > application
> > > > > > DAG (hence the name DAG).  I am proposing that we allow loops in
> > the
> > > > DAG
> > > > > if
> > > > > > the loop advances the window ID by a configured amount.  A JIRA
> > > ticket
> > > > > has
> > > > > > been created:
> > > > > >
> > > > > > https://malhar.atlassian.net/browse/APEX-60
> > > > > >
> > > > > > I have started this work in my fork at
> > > > > > https://github.com/davidyan74/incubator-apex-core/tree/APEX-60.
> > > > > >
> > > > > > The current progress is that a simple test case works.  Major
> work
> > > > still
> > > > > > needs to be done with respect to recovery and partitioning.
> > > > > >
> > > > > > The value ITERATION_WINDOW_COUNT is an attribute to an input port
> > of
> > > an
> > > > > > operator.  If the value of the attribute is greater than or equal
> > to
> > > 1,
> > > > > any
> > > > > > tuples sent to the input port are treated to be
> > > ITERATION_WINDOW_COUNT
> > > > > > windows ahead of what they are.
> > > > > >
> > > > > > For recovery, we will need to checkpoint all the tuples between
> > ports
> > > > > with
> > > > > > the to replay the looped tuples.  During the recovery, if the
> > > operator
> > > > > has
> > > > > > an input port, with ITERATION_WINDOW_COUNT=2, is recovering from
> > > > > checkpoint
> > > > > > window 14, the tuples for that input port from window 13 and
> window
> > > 14
> > > > > need
> > > > > > to be replayed to be treated as window 15 and window 16
> > respectively
> > > > > (13+2
> > > > > > and 14+2).
> > > > > >
> > > > > > In other words, we need to store all the tuples from window with
> ID
> > > > > > committedWindowId minus ITERATION_WINDOW_COUNT for recovery and
> > purge
> > > > the
> > > > > > tuples earlier than that window.
> > > > > > We can optimize this by only storing the tuples for
> > > > > ITERATION_WINDOW_COUNT
> > > > > > windows prior to any checkpoint.
> > > > > >
> > > > > > For that, we need a storage mechanism for the tuples.  Chandni
> > > already
> > > > > has
> > > > > > something that fits this usage case in Apex Malhar.  The class is
> > > > > > IdempotentStorageManager.  In order for this to be used in Apex
> > core,
> > > > we
> > > > > > need to deprecate the class in Apex Malhar and move it to Apex
> > Core.
> > > > > >
> > > > > > A JIRA ticket has been created for this particular work:
> > > > > >
> > > > > > https://malhar.atlassian.net/browse/APEX-128
> > > > > >
> > > > > > Some of the above has been discussed among Thomas, Chetan,
> Chandni,
> > > and
> > > > > > myself.
> > > > > >
> > > > > > For partitioning, we have not started any discussion or
> > > brainstorming.
> > > > > We
> > > > > > appreciate any feedback on this and any other aspect related to
> > > > > supporting
> > > > > > iterations in general.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > David
> > > > > >
> > > > >
> > > >
> > >
> >
>

--001a11c25e3c335d57051ff92c19--