Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.incubator.apache.org
Date: Thu, 31 Mar 2016 16:00:27 +0000 (UTC)
From: "Eugene Kirpichov (JIRA)" <jira@apache.org>
To: commits@beam.incubator.apache.org
Message-ID: <JIRA.12944550.1456419046000.101582.1459440027829@Atlassian.JIRA>
In-Reply-To: <JIRA.12944550.1456419046000@Atlassian.JIRA>
References: <JIRA.12944550.1456419046000@Atlassian.JIRA>
 <JIRA.12944550.1456419046152@arcas>
Subject: [jira] [Commented] (BEAM-68) Support for limiting parallelism of a
 step
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/BEAM-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220089#comment-15220089 ] 

Eugene Kirpichov commented on BEAM-68:
--------------------------------------

I re-duped BEAM-169 against BEAM-92.
- I agree that it won't work with empty K, however that should be relatively unlikely if the user has enough data that sharding it makes sense, and if their hash function is good; and in some sharded sink scenarios it may be desirable to not write empty shards.
- I'm not sure what you mean by "won't scale": individual shards have to be written sequentially, but they can be written in parallel with each other in this proposed implementation: dynamic rebalancing will separate the shard keys from each other.

> Support for limiting parallelism of a step
> ------------------------------------------
>
>                 Key: BEAM-68
>                 URL: https://issues.apache.org/jira/browse/BEAM-68
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Daniel Halperin
>
> Users may want to limit the parallelism of a step. Two classic uses cases are:
> - User wants to produce at most k files, so sets TextIO.Write.withNumShards(k).
> - External API only supports k QPS, so user sets a limit of k/(expected QPS/step) on the ParDo that makes the API call.
> Unfortunately, there is no way to do this effectively within the Beam model. A GroupByKey with exactly k keys will guarantee that only k elements are produced, but runners are free to break fusion in ways that each element may be processed in parallel later.
> To implement this functionaltiy, I believe we need to add this support to the Beam Model.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)