beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <>
Subject [jira] [Commented] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
Date Thu, 29 Jun 2017 18:23:00 GMT


Eugene Kirpichov commented on BEAM-2140:

So, to elaborate on what Kenn said. We dug a bit deeper into this yesterday and came up with
the following conclusions.

1) The reason that this stuff works in Dataflow and Direct runner is that, for running SDF,
they use a code path that simply _does not drop late data/timers or GC state_. These happen
in LateDataDroppingRunner and ReduceFnRunner and StatefulDoFnRunner - and the path for running
ProcessFn does not involve any of these. Aljoscha, maybe you can see why your current codepaths
for running ProcessFn in Flink involve dropping of late data / late timers, and make them
not involve it? :) (I'm not sure where this dropping happens in Flink)
2) As a consequence, however, state doesn't get GC'd. In practice this means that, if you
apply an SDF to input that is in many windows (e.g. to input windowed by fixed or sliding
windows), it will slowly leak state. However, in practice this is likely not a huge concern
because SDFs are expected to mostly be used when the amount of input is not super large (at
least compared to output), and it is usually globally windowed. Especially in streaming use
cases. I.e. it can be treated as a "Known issue" rather than "SDF does not work at all". *I
would recommend proceeding to implement it in Flink runner with this same known issue*, and
then solving the issue uniformly across all runners.

Posting this comment for now and writing another on how to do it without state leakage.

> Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
> -------------------------------------------------------
>                 Key: BEAM-2140
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
> As discovered as part of BEAM-1763, there is a failing SDF test. We disabled the tests
to unblock the open PR for BEAM-1763.

This message was sent by Atlassian JIRA

View raw message