beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <>
Subject [jira] [Created] (BEAM-1234) Consider a hint ParDo.withHighFanout()
Date Sat, 31 Dec 2016 17:33:58 GMT
Eugene Kirpichov created BEAM-1234:

             Summary: Consider a hint ParDo.withHighFanout()
                 Key: BEAM-1234
             Project: Beam
          Issue Type: Improvement
          Components: sdk-java-core
            Reporter: Eugene Kirpichov
            Assignee: Davor Bonaci
            Priority: Minor

I'm finding myself again and again suggesting users on StackOverflow to insert fusion breaks
after high-fanout ParDo's.

I think we should just implement this as a hint on ParDo and MapElements transforms, like
we have on GroupByKey.fewKeys() or Combine.withHotKeyFanout().

E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner that implements
fusion could decide to insert a runner-specific fusion break. This somewhat sidesteps the
issues in and
because every runner can decide how to do the right thing, or is free to ignore the hint.

This message was sent by Atlassian JIRA

View raw message