beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chamikaramj <...@git.apache.org>
Subject [GitHub] beam pull request #1890: [BEAM-564] Updates Python source API to allow repor...
Date Wed, 01 Feb 2017 19:53:33 GMT
GitHub user chamikaramj opened a pull request:

    https://github.com/apache/beam/pull/1890

    [BEAM-564] Updates Python source API to allow reporting consumed and remaining number
of split points

    With this update Python BoundedSource/RangeTracker API can report consumed and remaining
number of split points while performing a source read operations. Java SDK source API already
supports reporting these signals.
    
    These signals can be used by runner implementations, for example, to perform scaling decisions.
    
    This provides a slightly simplified API compared to previous PR #881.
    
    Main differences compared to #881 are following.
    
    (1) set_done()/done() methods were removed from the RangeTracker interface.
    
    Downside is that RangeTracker will be unable to provide the signal that all records have
been consumed. I think this signal is unnecessary since a runner can detect that anyways since
the reader loop of the source ends at that point.
    
    (2)
    Callback between BoundedSource and RangeTracker was changed from reporting remaining number
of split points to reporting unclaimed number of split points. This makes the implementation
of the callback simpler for source authors.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/chamikaramj/beam limited_parallelism_updated

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/1890.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1890
    
----
commit 8c5761f1647b11c9b60919bdecfa2ac77f4e491d
Author: Chamikara Jayalath <chamikara@google.com>
Date:   2017-01-05T03:10:09Z

    Updates Python SDK source API so that sources can report limited parallelism signals.
    
    With this update Python BoundedSource/RangeTracker API can report consumed and remaining
number of split points while performing a source read operations, similar to Java SDK sources.
    
    These signals can be used by runner implementations, for example, to perform scaling decisions.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message