flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Session Based Windows
Date Sat, 17 Oct 2015 12:39:45 GMT
Hi Paul,
it’s good to see people interested in this. I sketched a Trigger that should fit your requirements:

You can use it like this:

DataStream<> input = …
DataStream<> result = input
  .trigger(new SessionTrigger(timeout, maxElements))
  .apply(new MyWindowFunction())

The Trigger uses the new state API that I’m currently introducing in a new Pull Request.
It should be merged very soon, before the 0.10 release.

This implementation has one caveat, though. It cannot deal with elements that belong to different
sessions that arrive intermingled with other sessions. The reason is that Flink does not yet
support merging the windows that the WindowAssigner assigns as, for example, the Cloud Dataflow
API supports. This means that elements cannot be assigned to session windows, instead the
workaround with the GlobalWindow has to be used. I want to tackle this for the release after
0.10, however.

Please let us know if you need more information. I’m always happy to help in these interesting
cases at the bleeding edge of what is possible. :-)


> On 16 Oct 2015, at 19:36, Hamilton, Paul <paul.hamilton01@sap.com> wrote:
> Hi,
> I am attempting to make use of the new window APIs in streaming to
> implement a session based window and am not sure if the currently provided
> functionality handles my use case.  Specifically what I want to do is
> something conceptually similar to a ³Sessions.withGapDuration(Š)² window
> in Google DataFlow.
> Assuming the events are keyed by session id.  I would like to use the
> event time and the watermarking functionality to trigger a window after
> the ³end of a session² (no events for a given session received within x
> amount of time).  With watermarking this would mean trigger when a
> watermark is seen that is > (the time of the last event + session
> timeout). Also I want to perform an early triggering of the window after a
> given number of events have been received.
> Is it currently possible to do this with the current combination of window
> assigners and triggers?  I am happy to write custom triggers etc, but
> wanted to make sure it wasn¹t already available before going down that
> road.
> Thanks,
> Paul Hamilton
> Hybris Software

View raw message