Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
MIME-Version: 1.0
In-Reply-To: <CAC+nA48y9H+5T+eDMcw6B472EE4huiXq71O57Bi1k3Byo9xwzA@mail.gmail.com>
References: <CAC+nA48y9H+5T+eDMcw6B472EE4huiXq71O57Bi1k3Byo9xwzA@mail.gmail.com>
From: David Yan <david@datatorrent.com>
Date: Thu, 29 Sep 2016 16:29:44 -0700
Message-ID: <CAMqituODZ5=EAGu3NbsiZN6Zg-1=jUozCspsUFoVm+mx7pSgiA@mail.gmail.com>
Subject: Re: Retraction
To: dev@apex.apache.org
Content-Type: multipart/alternative; boundary=001a114a9b2e55a0cf053daddaa1
archived-at: Thu, 29 Sep 2016 23:29:57 -0000

--001a114a9b2e55a0cf053daddaa1
Content-Type: text/plain; charset=UTF-8

Hi Shunxin:

Yes, a retraction is just to tell downstream whatever the windowed operator
sent before was wrong.

In this blog (
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102), their
normal trigger value is an integer and the retraction trigger is just the
negative of the previous integer, and it assumes the downstream will
understand that, which it will if the downstream only does a delta
(addition or subtraction) to its state based on the incoming tuple, without
any regard to whether the trigger is a normal trigger or a retraction
trigger.

If the downstream needs to distinguish between normal and retraction
triggers, the user can override fireRetractionTrigger method.

But I think in a real use case, the downstream mostly likely can just
overwrite the old value with the new value so retraction is not needed. And
retraction is likely to be used mostly with session windows, because for
session windows, we need a way to tell downstream that some session windows
are not valid any more (result from session window merge or extension), and
I think exactly how retractions are handled is up to the application. For
now, I think it's fine for general purpose accumulations (like TopN or
ReduceN) to return null for the getRetraction method.

David

On Thu, Sep 29, 2016 at 3:56 PM, Shunxin Lu <lushunxin@gmail.com> wrote:

> Hi,
>
> I am planning to revisit the accumulation implementations and rewrite the
> getRetraction() method so that it behaves correctly. I understand that the
> concept of retraction is the upstream operator telling its downstream that
> the previous output is wrong and give it a new output. But it become
> confusing when it comes to actual implementation, how exactly should
> retraction be done in accumulations such as TopN and ReduceFn?
>
> Thanks!
> Shunxin Lu
>

--001a114a9b2e55a0cf053daddaa1--