beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <>
Subject [jira] [Updated] (BEAM-1164) Allow a DoFn to opt in to mutating it's input
Date Thu, 30 Mar 2017 17:22:41 GMT


Daniel Halperin updated BEAM-1164:
    Issue Type: New Feature  (was: Bug)

> Allow a DoFn to opt in to mutating it's input
> ---------------------------------------------
>                 Key: BEAM-1164
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Frances Perry
>            Priority: Minor
> Runners generally can't tell if a DoFn is mutating inputs, but assuming so by default
leads to significant performance implications from unnecessary copying (around sibling fusion,
etc). So instead the model prevents mutating inputs, and the Direct Runner validates this
behavior. (See:

> However, if users are processing a small number of large records by making incremental
changes (for example, genomics use cases), the cost of immutability requirement can be very
large. As a workaround, users sometimes do suboptimal things (fusing ParDos by hand) or undefined
things when they expect the immutability requirement is unnecessarily strict (adding no-op
coders in places they hope the runner won't be materializing things, mutating things anyway
when they don't expect sibling fusion to happen, etc).
> We should consider adding a signal (MutatingDoFn?) that users explicitly opt in to to
say their code may mutate inputs. The runner can then use this assumption to either prevent
optimizations that would break in the face of this or insert additional copies as needed to
allow optimizations to preserve semantics.
> See this related user@ discussion:

This message was sent by Atlassian JIRA

View raw message