beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kennknowles <>
Subject [GitHub] beam pull request #3750: [BEAM-2430] DoFnRunner for processing a bundle over...
Date Wed, 23 Aug 2017 14:57:27 GMT
GitHub user kennknowles opened a pull request:

    [BEAM-2430] DoFnRunner for processing a bundle over the Fn API

    Follow this checklist to help us incorporate your contribution quickly and easily:
     - [ ] Make sure there is a [JIRA issue](
filed for the change (usually before you start working on it).  Trivial changes like typos
do not require a JIRA issue.  Your pull request should address just this issue, without pulling
in other changes.
     - [ ] Each commit in the pull request should have a meaningful subject line and body.
     - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`,
where you replace `BEAM-XXX` with the appropriate JIRA issue.
     - [ ] Write a pull request description that is detailed enough to understand what the
pull request does, how, and why.
     - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
     - [ ] If this contribution is large, please file an Apache [Individual Contributor License
    R: @aljoscha this is not super polished, and much of the complexity I thought I would
encapsulate really fits better outside the DoFnRunner. But I thought reviewing this and getting
it in might be a good way to proceed and share understanding of the Fn API.

You can merge this pull request into a Git repository by running:

    $ git pull FnApiDoFnRunner

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3750
commit eb18585969f67089de62b20dc5c3b2722d7e566a
Author: Kenneth Knowles <>
Date:   2017-08-23T13:20:33Z

    Basic Java wrapper for Fn API data plane

commit f71101685265c53ecee36d3aadec037cc792f0b4
Author: Kenneth Knowles <>
Date:   2017-07-26T14:42:03Z

    FnApiControlClient, de-inverts Fn API control plane
    The Fn API control plane has an inverted client/server relationship in
    order to support firewall rules where the runner is forbidden from
    connecting out to the SDK harness. This Java helper provides an API with
    the more conventional polarity. It also associates streamed gRPC
    requests with responses to support simple future-based programming.

commit 858f219ce3a6df64b8344472022e01141ecf130b
Author: Kenneth Knowles <>
Date:   2017-07-26T14:44:41Z

    Service for pooling incoming Fn API control plane connections

commit c3c882df800926b46c07bdeb867b5639bcde00f1
Author: Kenneth Knowles <>
Date:   2017-07-26T14:45:37Z

    Higher-level SdkHarnessClient
    This adds a fairly thin wrapper on FnApiSdkHarnessClient, encapsulating
    the fact that all request and response types are injected into a
    disjoint union and sent over the same low-level RPC.

commit a947ccb752f4200cbc8a98f85129115e45577973
Author: Kenneth Knowles <>
Date:   2017-07-26T14:48:48Z

    Add SdkHarnessDoFnRunner
    This encapsulates processing a bundle over the Fn API.
    A Beam runner is responsible for:
     - Setting up a FnApiControlClientPoolService to listen for incoming
       FnApiSdkHarnessClient connections
     - Wrapping those connections in the higher-level SdkHarnessClient
     - Building the ProcessBundleDescriptor (instruction graph) to be executed
     - Establishing data plane endpoints referenced by the ProcessBundleDescriptor
     - Registering the data plane endpoints and ProcessBundleDescriptor with the
    This class is responsible for:
     - Registering each bundle with the SDK harness
     - Streaming the elements of each bundle to the SDK harness
       over the data plane


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message