flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Wilson <paulalexwil...@gmail.com>
Subject Re: Custom Barrier?
Date Tue, 14 Jun 2016 13:04:21 GMT

No these super-structure events only serve the purpose of defining the
boundaries of a join, and do not relate to the keys of the sub-events.


On 14 June 2016 at 10:32, Aljoscha Krettek <aljoscha@apache.org> wrote:

> Hi,
> would these super-structure events occur per key? If yes, then I think you
> can process this using the currently available windowing mechanism by
> writing a custom WindowAssigner and Trigger. This, of course, assumes that
> the events arrive in-order, i.e. if A-End arrives before A-Start or if
> elements that should fall inside the A window arrive after A-End then I
> don't see an easy way to do it.
> Let me know if you need to know more about assigners/triggers.
> Cheers,
> Aljoscha
> On Mon, 13 Jun 2016 at 16:29 Paul Wilson <paulalexwilson@gmail.com> wrote:
>> Hi,
>> I've been evaluating Flink and wondering if it was possible to define a
>> window that is based on characteristics of the data (data driven) but not
>> contained in the data stream directly.
>> Consider 'nested events' where lower level events belong to a wider event
>> where the wider event serves only to define a boundary (or window) over the
>> lower level events. I was wondering if there was some way to communicate
>> this super-structure in the stream somehow?
>> I know that Flink users 'barriers' to define snapshot boundaries, but it
>> might it be possible to communicate a 'window end' in a similar fashion?
>> I guess I could attach an additional value to each event using a stateful
>> map function and then define the window on that?
>> e.g. A-Start, 1, 2, 3, A-End, B-Start, 1, 2, 3, B-End
>> Regards,
>> Paul

View raw message