beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <>
Subject [jira] [Created] (BEAM-801) SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
Date Sun, 23 Oct 2016 22:02:58 GMT
Kenneth Knowles created BEAM-801:

             Summary: SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
                 Key: BEAM-801
             Project: Beam
          Issue Type: Bug
          Components: runner-core, runner-direct
            Reporter: Kenneth Knowles

Today, runners-core contains an expansion of {{ParDo(DoFn)}} that implements it via unsupported
features of {{ParDo(OldDoFn)}}. This expansion is used by the DirectRunner.

The right approach to provide a ready implementation in runners-core is the one taken by {{GroupAlsoByWindowViaWindowSetDoFn}}
where the unsupported features are provided to the constructor of the {{DoFn}}, rather than
expected to be passed through the {{ProcessContext}}.

These features are not guaranteed to be part of the public state & timers API of {{DoFn}},
particularly in the case of watermark holds, so it is not prudent to plan on waiting for the
new state & timers API and porting over.

These issues create real blockers by causing dependency cycles between implementing {{DoFn}}
fully (requires no use of unsupported features), implementing new state and timers (requires
new {{DoFn}}), and keeping the hack until new {{DoFn}} has state and timers (requires use
of unsupported features).

To break the cycle, the tests that rely on unsupported features are being disabled. They can
be re-enabled either by following the design suggested above (probably the most robust approach)
or by waiting and porting to new features as they are available.

This message was sent by Atlassian JIRA

View raw message