beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jkff <>
Subject [GitHub] beam pull request #3607: [BEAM-2512] Introduces
Date Fri, 21 Jul 2017 00:38:17 GMT
GitHub user jkff opened a pull request:

    [BEAM-2512] Introduces
    Part of, based on
    This PR includes - reviewer should look only
at the other commit. Also, requires to properly support
read().from(ValueProvider).watchForNewFiles() - this PR should be submitted only after both
of the PRs above.
    R: @reuvenlax 

You can merge this pull request into a Git repository by running:

    $ git pull textio-read-watch-new-files

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3607
commit 3103f9438a9fc392dfa1c37ceac990fc43c2ab98
Author: Eugene Kirpichov <>
Date:   2017-07-20T02:50:03Z

    [BEAM-2623] Introduces Watch transform
    The transform watches for new elements in a family of growing sets.
    See design at
    As part of the implementation, I found and fixed a bug in tracking the
    watermark in OutputAndTimeBoundedSplittableProcessElementInvoker.
    The watermark must be captured at the moment checkpoint is taken,
    because it describes timestamps of elements output from the checkpoint.
    I also made direct runner by default checkpoint SDF's every 100 elements
    rather than every 10000, to make it more aggressive - that's what
    uncovered the bug above.

commit c977d606f14a59ed73acf22f32a6b250d89c0ccd
Author: Eugene Kirpichov <>
Date:   2017-07-20T23:58:42Z

    [BEAM-2512] Introduces
    Part of, based on


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message