beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Shields (JIRA)" <>
Subject [jira] [Commented] (BEAM-175) Leak garbage collection timers in GlobalWindow
Date Thu, 12 May 2016 19:09:12 GMT


Mark Shields commented on BEAM-175:

Here's the proposed solution, which pull/139 prototypes:
(Note that pull/139 also fixes and
since they are all in the same space.)

- Window.ClosingBehavior becomes Window.EmptyPaneBehavior
- New property WindowingStrategy.onTimeBehavior of type Window.EmptyPaneBehavior, which is
the analogue of closingBehavior but for the ON_TIME pane. Default is FIRE_ALWAYS.
- New enum Window.PaneIndexBehavior { SEQUENTIAL, ZERO }
- New property WindowingStrategy.paneIndexBehavior of type Window.PaneIndexBehavior which
controls how PaneInfo.getIndex and friends are determined. Default is SEQUENTIAL.
- In ReduceFnRunner.emitIfApproriate we delete any outstanding end-of-window or garbage-collection
timer if we can be sure the corresponding onTimer would be a no-op. This means:
    - No element state, either combiner or iterable.
    - No end-of-window/garbage-collection holds, which will have been elided if the onTimeBehavior/closingBehavor
    - No trigger finished bits (ie the trigger must be a Repeatably which never closes).
    - No previous pane info, which will have been elided if paneIndexBehavior is ZERO.
    - No active window state, ie the windows are not merging or there's no currently active
merging window.

The upshot is the following classic pattern will never leave behind state per key:
  Window.<KV<Long, Void>>into(new GlobalWindows())

> Leak garbage collection timers in GlobalWindow
> ----------------------------------------------
>                 Key: BEAM-175
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-core
>            Reporter: Mark Shields
>            Assignee: Mark Shields
> Consider the  transform:
>   Window
>     .into(new GlobalWindows())
>     .triggering(
>       Repeatedly.forever(
>         AfterProcessingTime.pastFirstElementInPane().plusDelayOf(...)))
>     .discardingFiredPanes()
> This is a common idiom for 'process elements bunched by arrival time'.
> Currently we create an end-of-window timer per key, which clearly will only fire if the
pipeline is drained.
> Better would be to avoid creating end-of-window timers if there's no state which needs
to be processed at end-of-window (ie at drain if the Global window).

This message was sent by Atlassian JIRA

View raw message