beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kenneth Knowles <...@google.com>
Subject Re: Event-time based in-window trigger
Date Fri, 02 Dec 2016 03:40:58 GMT
Thanks for laying out some details.

On Thu, Dec 1, 2016 at 7:09 PM, Manu Zhang <owenzhang1990@gmail.com> wrote:
>
> Yes, the difficulty is to define that trigger. The existing triggers fire
> at the end of window. (I could be mistaken, which will be good news)
>

You are not mistaken that the only existing event time trigger is the one
that fires at the end of the window. The trigger you describe would be a
new primitive trigger. It fits with the design, if we ensure monotonicity,
etc. Actually implementing it in the backend is easy, of course. We
actually had something like it, but didn't quite nail it down so we removed
it until we had a solid use case and design for it.

B and C which are not mutually exclusive
> More on my use case. Say a user visits http://foo at 1, http://foo/bar at
> 4 and back to http://foo at 5 all in a Session
> we would want to emit
>
> http://foo  when the watermark passes 1
> http://foo -> http://foo/bar when the watermark passes 4
> http://foo -> http://foo/bar -> http://foo when the watermark passes 5
>

What would you want to emit when the watermark jumps from 0 to 7 and all
three of the above are buffered?

What would you want to emit when the watermark was at 9 and
http://foo/bizzle came in with timestamp 3?

Kenn


>
>
> On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <bchambers@google.com> wrote:
>
>> As a clarifying question:
>>
>> If you have three elements in the pane with timestamps [1, 4, 5], would
>> you:
>> A. want to emit that entire pane when the watermark passes 1
>> B. want to emit that entire pane when the watermark passes 5
>> C. emit a fragment of that pane containing only the first element when
>> the watermark passes 1
>>
>> On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <takidau@google.com> wrote:
>>
>> So what you want is essentially a trigger that fires when the watermark
>> has passed the event time of the oldest un-emitted element in the current
>> pane? You could them presumably wrap this in a repeat to get the overall
>> desired semantics, right?
>>
>> -Tyler
>>
>>
>> On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <owenzhang1990@gmail.com>
>> wrote:
>>
>> My use case is to track user trajectory based on page view event when
>> they visit a website.  The input would be like a list of
>> PageView(userId, url, eventTimestamp) with watermarks (= eventTimestamp -
>> duration). I'm trying Sessions with event time trigger. Note we can't wait
>> for the end of session window due to latency requirement. Instead, we want
>> to emit the user trajectories whenever a buffered PageView's event time is
>> passed by watermark.
>>
>> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lcwik@google.com> wrote:
>>
>> Can you provide more details about the problem your trying to solve with
>> some examples showing input and the expected output?
>>
>>
>>
>>
>> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <owenzhang1990@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> Recently I’m addressing a problem where users want to trigger after
>> watermark past each element (i.e. in the middle of event-time window). I
>> fail to find an existing trigger that does so. Any idea on model this
>> problem with Beam ?
>>
>> Thanks,
>> Manu Zhang
>>
>>
>>

Mime
View raw message