beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BEAM-1234) Consider a hint ParDo.withHighFanout()
Date Wed, 19 Jul 2017 20:21:02 GMT


ASF GitHub Bot commented on BEAM-1234:

Github user asfgit closed the pull request at:

> Consider a hint ParDo.withHighFanout()
> --------------------------------------
>                 Key: BEAM-1234
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-core
>            Reporter: Eugene Kirpichov
>            Priority: Minor
> I'm finding myself again and again suggesting users on StackOverflow to insert fusion
breaks after high-fanout ParDo's.
> I think we should just implement this as a hint on ParDo and MapElements transforms,
like we have on GroupByKey.fewKeys() or Combine.withHotKeyFanout().
> E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner that implements
fusion could decide to insert a runner-specific fusion break. This somewhat sidesteps the
issues in and
because every runner can decide how to do the right thing, or is free to ignore the hint.

This message was sent by Atlassian JIRA

View raw message