beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From XuMingmin <...@git.apache.org>
Subject [GitHub] beam pull request #3603: redo PR 3553
Date Thu, 20 Jul 2017 17:11:09 GMT
GitHub user XuMingmin opened a pull request:

    https://github.com/apache/beam/pull/3603

    redo PR 3553

    Follow this checklist to help us incorporate your contribution quickly and easily:
    
     - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/)
filed for the change (usually before you start working on it).  Trivial changes like typos
do not require a JIRA issue.  Your pull request should address just this issue, without pulling
in other changes.
     - [ ] Each commit in the pull request should have a meaningful subject line and body.
     - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`,
where you replace `BEAM-XXX` with the appropriate JIRA issue.
     - [ ] Write a pull request description that is detailed enough to understand what the
pull request does, how, and why.
     - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
     - [ ] If this contribution is large, please file an Apache [Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.pdf).
    
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apache/beam master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3603.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3603
    
----
commit 8cab15338f811f880c6cfb820051cf355f92986b
Author: Robert Bradshaw <robertwb@google.com>
Date:   2017-06-22T01:09:48Z

    Java Dataflow runner harness compatibility.

commit 2a5520018fb25f96b57e52231a8cc0b191abfce0
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-06-22T19:34:27Z

    Closes #3418

commit 7471e2736cc22336500f6252ab8448889a2d04d3
Author: Charles Chen <ccy@google.com>
Date:   2017-06-22T18:29:54Z

    Clean up test_stream_test and remove stray print statement

commit 3dc454a9bab4683efea080a1cd5484f0838561d3
Author: Ahmet Altay <altay@google.com>
Date:   2017-06-22T20:17:50Z

    This closes #3423

commit 0292a24f9c88796542bff55031d84c11f0ab6b16
Author: Colin Phipps <fipsy@google.com>
Date:   2017-05-15T14:18:16Z

    [BEAM-2439] Dynamic sizing of Datastore write RPCs
    
    This stops the Datastore connector from always sending 500 entities per RPC.
    Instead, it starts at a lower number which is more likely to complete within
    the deadline even in adverse conditions, and then increases or reduces the
    batch size in response to measured latency of past requests.

commit 9ed0af8f2c5dabf3af5d664c473a49166b96d7d2
Author: Luke Cwik <lcwik@google.com>
Date:   2017-06-22T20:34:12Z

    [BEAM-2439] Dynamic sizing of Datastore write RPCs
    
    This closes #3390

commit c05764454e73ab93d0602f34b8b7622d46e1d892
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T03:58:35Z

    DataflowRunner: Reject SetState and MapState

commit 5506be87dc3b8f4c4425a259cf255a168fd7b127
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T20:51:46Z

    This closes #3420: DataflowRunner: Reject SetState and MapState

commit 497cfabea7d6dcee0c5d327022678c571c3ec487
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T18:31:28Z

    Add window matcher for pane info

commit d4e5db51a025a831ddf4e3bc0e003caebabf647b
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T18:56:53Z

    Tidy LateDataDroppingDoFnRunner

commit 1c1f239501349f5120b0d619c4eea9c435500b78
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T19:52:42Z

    ReduceFnTester can advance clocks without firing timers

commit 795760d370bcbe28e1f0ca373ad4c8c841e6e6b5
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T19:53:15Z

    ReduceFnTester assertion for windows that have data buffered

commit 412fd7eab9e58a4d412f4dff5ffec023610b4f22
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T19:56:14Z

    Drop late data in ReduceFnTester

commit 50c43d96adb8c2523cf38c09f32e241eacc47823
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T19:56:34Z

    Do not GC windows based on processing time timer!

commit fda589c00c8920e76cfc9aaa87cecfa94077599d
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T20:04:23Z

    Add test reproducing BEAM-2505, ignored

commit d2b384a20dbb0213d0f63e74713a06d63bad8d39
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T20:05:42Z

    Add tests for corner cases of processing time timers

commit 7645c44b9bae9a77192fbe6744918cfb95ca1a3a
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T22:41:02Z

    This closes #3425: Fix processing time timer handling in ReduceFnRunner
    
      Add tests for corner cases of processing time timers
      Add test reproducing BEAM-2505, ignored
      Do not GC windows based on processing time timer!
      Drop late data in ReduceFnTester
      ReduceFnTester assertion for windows that have data buffered
      ReduceFnTester can advance clocks without firing timers
      Tidy LateDataDroppingDoFnRunner
      Add window matcher for pane info

commit 5d6ad19958d0a2394f9e33720a04cc954279a7e7
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-06-22T19:44:23Z

    Remove fn api bundle descriptor translation.

commit a882e8f3a33c4a430f55d53b65285123c5a4f50d
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-06-22T19:46:13Z

    Remove unused (and untested) initial splittling logic.

commit 7a9f762e440168d48e3b3b2930dd83885402e51a
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-06-23T00:05:32Z

    Closes #3424

commit 799173fac4e07dab4547ba21971922336cd72c62
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-06-21T22:21:32Z

    Uses KV in SplittableParDo expansion instead of ElementAndRestriction
    
    This is a workaround for the following issue.
    
    ElementAndRestriction is in runners-core, which may be shaded by runners
    (and is shaded by Dataflow runner), hence it should be *both* produced
    and consumed by workers - but currently it's produced by (shaded)
    SplittableParDo and consumed by (differently shaded) ProcessFn in the
    runner's worker code.
    
    There are several ways out of this, e.g. moving EAR into the SDK (icky
    because it's an implementation detail of SplittableParDo), or using
    a type that's already in the SDK. There may be other more complicated
    ways too.

commit 6f12e7d3d6a9fbd0d7bc1a6136542bd503cb0f2b
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-06-22T21:59:36Z

    Bump Dataflow worker to 0622

commit 8860cceb710f28931fe5ec792b0db09277f0d2a9
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-06-23T00:56:44Z

    This closes #3417: [BEAM-1377] Uses KV in SplittableParDo expansion instead of ElementAndRestriction

commit cab4d8969e7f95b0ece59838ad2d578e75d38823
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-22T03:25:31Z

    DataflowRunner: Reject merging windowing for stateful ParDo

commit 5f8cfa7412e8f4ce97d6df4413fecbea22b4b5e1
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-23T03:05:28Z

    This closes #3419: DataflowRunner: Reject merging windowing for stateful ParDo

commit 649994b353afe28c917969609c7a1a47a4f39aaf
Author: Rune Fevang <fevang@exabel.com>
Date:   2017-06-15T11:51:12Z

    Allow output from FinishBundle in DoFnTester

commit b3c36256e32f4a4a616d0e152686d29f1174c5cd
Author: Kenneth Knowles <klk@google.com>
Date:   2017-06-23T03:07:48Z

    This closes #3367: Allow output from FinishBundle in DoFnTester

commit 8dcda6e40355af13f4d92fcd44aae4539a225a4a
Author: Sourabh Bajaj <sourabhbajaj@google.com>
Date:   2017-06-23T00:08:20Z

    [BEAM-2497] Fix the reading of concat gzip files

commit 336b7f1cffc8709cd480a243a16de95bce39b53e
Author: Chamikara Jayalath <chamikara@google.com>
Date:   2017-06-23T05:13:05Z

    This closes #3428

commit f291713b28e3ba0246a8c0a710c71506cd0a0f91
Author: Etienne Chauchot <echauchot@gmail.com>
Date:   2017-06-21T08:39:39Z

    [BEAM-2489] Use dynamic ES port in HIFIOWithElasticTest

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message