Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
From: Aljoscha Krettek <aljoscha@apache.org>
Date: Wed, 09 Nov 2016 13:22:51 +0000
Message-ID: <CANMXwW1mE9E5b380WQ39xrkQwh-_1dhak+vRL72yMU+VLhe=mg@mail.gmail.com>
Subject: [DISCUSS] Changing Window Cleanup Semantics
To: Dev <dev@flink.apache.org>, user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c18a3de0405750540de28e8
archived-at: Wed, 09 Nov 2016 13:23:05 -0000

--94eb2c18a3de0405750540de28e8
Content-Type: text/plain; charset=UTF-8

Hi,
I recently created https://issues.apache.org/jira/browse/FLINK-4994 to
address what I think is a flaw in the window cleanup semantics. This has
the possibility of affecting people so I'd like to get some opinions and
also give people a heads-up.

Before going into what I'm proposing in the issue, let's first see how
window cleanup works in Flink 1.1. There are three pieces of state that
need to be cleaned up: 1) window contents (the actual elements) 2) merging
window set (only in case of merging windows) and 3) trigger state. The
WindowOperator is responsible for cleaning the first two while the Trigger
itself is responsible for the third. For this purpose we have the method
Trigger.clear() that must clean up all state the Trigger created.

In Flink 1.1 we clear 1 and 2 and call Trigger.clear() when either of two
things happens: a Trigger returns PURGE (or FIRE_AND_PURGE, below I'll only
say PURGE but mean both) or the allowed lateness expires (watermark passes
the end of the window plus allowed lateness). This will lead to surprising
results when you use session windows (as outlined in the linked issue) and
people also seem to have been working around the fact that Trigger.clear()
is called when PURGING because they want to clear the window contents but
still keep some state in the trigger in case more elements arrive.

For Flink 1.2 I'm proposing to not clear the merging window set on PURGE
and also to not call Trigger.clear() on PURGE. Those two should only happen
when we garbage collect the window, i.e. we reach the end of the allowed
lateness. This will have the consequence that people who have been relying
on cleanup to happen after a PURGE have to manually cleanup state now. I
think it's still a good change, though, since it leads to more sane
behaviour for merging windows.

What do you think?

There is also the possibility of adding a Trigger.onPurge() method that by
default calls clean(), but in the long run I'd like to remove that again so
at one point people have to adapt to the changes. In my opinion it's better
to break this now and cleanly.

Cheers,
Aljoscha

--94eb2c18a3de0405750540de28e8--