beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ben Chambers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-175) Leak garbage collection timers in GlobalWindow
Date Mon, 16 May 2016 15:57:12 GMT

    [ https://issues.apache.org/jira/browse/BEAM-175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284759#comment-15284759
] 

Ben Chambers commented on BEAM-175:
-----------------------------------

I think in general this seems OK. I like that we're making the behavior explicit rather than
trying to guessi t.

1. This may be too much configuration. There may be other/better ways of getting pane indices
(eg., once we have a sink API or a state API). We should make sure we understand the use cases
before exposing the knob. Especially since ReduceFnRunner is already complicated -- this is
just adding more permutations of cases it needs to handle.
2. I worry that the default, at least in the Pane Index case, is actually *less* performant
than the ZERO case, which is likely what was desired 90% of the time. If we go this direction,
I would propose we change the default.
3. You should flesh this out to address error cases. When do we detect that the user is accessing
the PaneIndex with the ZERO behavior specified. What kind of error message do they get? Etc.

> Leak garbage collection timers in GlobalWindow
> ----------------------------------------------
>
>                 Key: BEAM-175
>                 URL: https://issues.apache.org/jira/browse/BEAM-175
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-core
>            Reporter: Mark Shields
>            Assignee: Mark Shields
>
> Consider the  transform:
>   Window
>     .into(new GlobalWindows())
>     .triggering(
>       Repeatedly.forever(
>         AfterProcessingTime.pastFirstElementInPane().plusDelayOf(...)))
>     .discardingFiredPanes()
> This is a common idiom for 'process elements bunched by arrival time'.
> Currently we create an end-of-window timer per key, which clearly will only fire if the
pipeline is drained.
> Better would be to avoid creating end-of-window timers if there's no state which needs
to be processed at end-of-window (ie at drain if the Global window).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message