beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <>
Subject [jira] [Created] (BEAM-47) UnboundedSource should support dynamic work rebalancing
Date Thu, 25 Feb 2016 14:29:18 GMT
Daniel Halperin created BEAM-47:

             Summary: UnboundedSource should support dynamic work rebalancing
                 Key: BEAM-47
             Project: Beam
          Issue Type: New Feature
          Components: sdk-java-extensions
            Reporter: Daniel Halperin

BoundedSource has splitAtFraction and related code to enable large work items to be split
dynamically and moved to other workers.

UnboundedSource does not have this support -- instead, runners are expected to allocate many
splits initially and rebalance those between workers.

We should add dynamic work rebalancing to UnboundedSource. This will almost certainly be done
by checkpointing the source, and then the dynamically produced splits will both inherit the
checkpoint (or a split-adapted version of it).

This message was sent by Atlassian JIRA

View raw message