beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-1234) Consider a hint ParDo.withHighFanout()
Date Sat, 31 Dec 2016 17:33:58 GMT
Eugene Kirpichov created BEAM-1234:
--------------------------------------

             Summary: Consider a hint ParDo.withHighFanout()
                 Key: BEAM-1234
                 URL: https://issues.apache.org/jira/browse/BEAM-1234
             Project: Beam
          Issue Type: Improvement
          Components: sdk-java-core
            Reporter: Eugene Kirpichov
            Assignee: Davor Bonaci
            Priority: Minor


I'm finding myself again and again suggesting users on StackOverflow to insert fusion breaks
after high-fanout ParDo's.

I think we should just implement this as a hint on ParDo and MapElements transforms, like
we have on GroupByKey.fewKeys() or Combine.withHotKeyFanout().

E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner that implements
fusion could decide to insert a runner-specific fusion break. This somewhat sidesteps the
issues in https://issues.apache.org/jira/browse/BEAM-730 and https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
because every runner can decide how to do the right thing, or is free to ignore the hint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message