beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jkff <...@git.apache.org>
Subject [GitHub] beam pull request #4124: [BEAM-3169] Fixes a data loss bug in WriteFiles whe...
Date Tue, 14 Nov 2017 00:07:20 GMT
GitHub user jkff opened a pull request:

    https://github.com/apache/beam/pull/4124

    [BEAM-3169] Fixes a data loss bug in WriteFiles when used with fire-once triggers

    https://issues.apache.org/jira/browse/BEAM-3169
    
    This required a bit of twiddling with shard assignment logic too. The gist of the change
is changing the pre-finalize GBK to Reshuffle. I audited all other usages of GBK in the SDK
and it appears that only this one is buggy: others either explicitly set a repeated trigger
before applying the GBK, or are directly applied to the user's input and the user's trigger
firing behavior is WAI.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jkff/incubator-beam write-files-data-loss

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/4124.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4124
    
----
commit 17c4af7c5a3a9475a284afd4fe413cb57016e7e5
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-11-13T19:09:01Z

    Slightly simplifies fixed sharding

commit 3926abf84c681f3b6918a3d691ca2cb00a2e40e6
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-11-13T19:58:11Z

    More clear and consistent shard number assignment logic

commit e52d3d8d49d6ff2b2b767f8069e1f71f8c42c9ae
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-11-13T20:28:34Z

    Materializes file results via Reshuffle rather than GBK

commit 1a625ecf313b6ff03311464d40a5515736cbbdd7
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-11-13T23:55:44Z

    Adds test for WriteFiles with a fire-once trigger

commit cba9ca163c621c1b187965226748596e2f5f8600
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-11-14T00:04:03Z

    makes checkstyle happy

----


---

Mime
View raw message