apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Yan <da...@datatorrent.com>
Subject Re: Retraction
Date Thu, 29 Sep 2016 23:29:44 GMT
Hi Shunxin:

Yes, a retraction is just to tell downstream whatever the windowed operator
sent before was wrong.

In this blog (
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102), their
normal trigger value is an integer and the retraction trigger is just the
negative of the previous integer, and it assumes the downstream will
understand that, which it will if the downstream only does a delta
(addition or subtraction) to its state based on the incoming tuple, without
any regard to whether the trigger is a normal trigger or a retraction
trigger.

If the downstream needs to distinguish between normal and retraction
triggers, the user can override fireRetractionTrigger method.

But I think in a real use case, the downstream mostly likely can just
overwrite the old value with the new value so retraction is not needed. And
retraction is likely to be used mostly with session windows, because for
session windows, we need a way to tell downstream that some session windows
are not valid any more (result from session window merge or extension), and
I think exactly how retractions are handled is up to the application. For
now, I think it's fine for general purpose accumulations (like TopN or
ReduceN) to return null for the getRetraction method.

David

On Thu, Sep 29, 2016 at 3:56 PM, Shunxin Lu <lushunxin@gmail.com> wrote:

> Hi,
>
> I am planning to revisit the accumulation implementations and rewrite the
> getRetraction() method so that it behaves correctly. I understand that the
> concept of retraction is the upstream operator telling its downstream that
> the previous output is wrong and give it a new output. But it become
> confusing when it comes to actual implementation, how exactly should
> retraction be done in accumulations such as TopN and ReduceFn?
>
> Thanks!
> Shunxin Lu
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message