beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections
Date Tue, 10 May 2016 20:38:13 GMT


ASF GitHub Bot commented on BEAM-22:

GitHub user tgroh opened a pull request:

    [BEAM-22] Enable RunnableOnService Tests

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
     - [ ] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](
    Not ready for review. Publishing PR to hook into Travis and Jenkins.
    Update runners/direct-java/pom.xml to enable the RunnableOnService
    tests phase.

You can merge this pull request into a Git repository by running:

    $ git pull enable_ros_tests

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #319
commit d1796ba6fecb8423e563fcdf66946beda79e52c6
Author: Thomas Groh <>
Date:   2016-05-09T22:47:27Z

    Minor checkArgument style fix

commit f0e38fd170949f27d4794113e4bcb2077ffe88a6
Author: Thomas Groh <>
Date:   2016-05-10T18:27:37Z

    Use an AtomicReference in InProcessSideInputContainer
    This fixes a TOCTOU race in the contents updating logic, where the
    determination that the current pane should replace the contents of the
    side input and the replacement is not a single atomic operation. Using
    AtomicReference allows the use of compareAndSet to ensure that the
    replacement can only occur on the pane that the decision to replace was
    made with.
    Fixes a race where a pane could be the latest, and replace a
    pane, but would be lost due to an earlier pane being written between the
    invalidation and loading of contents.
    Fixes a race where a reader can incorrectly read an empty iterable as
    the contents of a PCollectionView, due to occuring between the
    invalidate and reload steps.

commit e06e449e3762a48404d0407babaff440ebfa416e
Author: Thomas Groh <>
Date:   2016-05-10T20:22:20Z

    Cache read SideInput Contents in the InProcessSideInputContainer
    This ensures that while processing a bundle all elements see the same
    contents for any SideInput Window.

commit 8ff1d79474f3d114381b924fa61aa46bd7b935db
Author: Thomas Groh <>
Date:   2016-05-10T20:36:21Z

    Enable RunnableOnService tests for the Direct Runner


> DirectPipelineRunner: support for unbounded collections
> -------------------------------------------------------
>                 Key: BEAM-22
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-direct
>            Reporter: Davor Bonaci
>            Assignee: Thomas Groh
> DirectPipelineRunner currently runs over bounded PCollections only, and implements only
a portion of the Beam Model.
> We should improve it to faithfully implement the full Beam Model, such as add ability
to run over unbounded PCollections, and better resemble execution model in a distributed system.
> This further enables features such as a testing source which may simulate late data and
test triggers in the pipeline. Finally, we may want to expose an option to select between
"debug" (single threaded), "chaos monkey" (test as many model requirements as possible), and
"performance" (multi-threaded).
> more testing (chaos monkey) 
> Once this is done, we should update this StackOverflow question:

This message was sent by Atlassian JIRA

View raw message