Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5D824200CCB for ; Thu, 20 Jul 2017 19:11:11 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 5BB501630F0; Thu, 20 Jul 2017 17:11:11 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7A565162B62 for ; Thu, 20 Jul 2017 19:11:10 +0200 (CEST) Received: (qmail 30460 invoked by uid 500); 20 Jul 2017 17:11:09 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 30446 invoked by uid 99); 20 Jul 2017 17:11:09 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jul 2017 17:11:09 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 8B3B4DFF8A; Thu, 20 Jul 2017 17:11:09 +0000 (UTC) From: XuMingmin To: commits@beam.apache.org Reply-To: commits@beam.apache.org Message-ID: Subject: [GitHub] beam pull request #3603: redo PR 3553 Content-Type: text/plain Date: Thu, 20 Jul 2017 17:11:09 +0000 (UTC) archived-at: Thu, 20 Jul 2017 17:11:11 -0000 GitHub user XuMingmin opened a pull request: https://github.com/apache/beam/pull/3603 redo PR 3553 Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/beam master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3603.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3603 ---- commit 8cab15338f811f880c6cfb820051cf355f92986b Author: Robert Bradshaw Date: 2017-06-22T01:09:48Z Java Dataflow runner harness compatibility. commit 2a5520018fb25f96b57e52231a8cc0b191abfce0 Author: Robert Bradshaw Date: 2017-06-22T19:34:27Z Closes #3418 commit 7471e2736cc22336500f6252ab8448889a2d04d3 Author: Charles Chen Date: 2017-06-22T18:29:54Z Clean up test_stream_test and remove stray print statement commit 3dc454a9bab4683efea080a1cd5484f0838561d3 Author: Ahmet Altay Date: 2017-06-22T20:17:50Z This closes #3423 commit 0292a24f9c88796542bff55031d84c11f0ab6b16 Author: Colin Phipps Date: 2017-05-15T14:18:16Z [BEAM-2439] Dynamic sizing of Datastore write RPCs This stops the Datastore connector from always sending 500 entities per RPC. Instead, it starts at a lower number which is more likely to complete within the deadline even in adverse conditions, and then increases or reduces the batch size in response to measured latency of past requests. commit 9ed0af8f2c5dabf3af5d664c473a49166b96d7d2 Author: Luke Cwik Date: 2017-06-22T20:34:12Z [BEAM-2439] Dynamic sizing of Datastore write RPCs This closes #3390 commit c05764454e73ab93d0602f34b8b7622d46e1d892 Author: Kenneth Knowles Date: 2017-06-22T03:58:35Z DataflowRunner: Reject SetState and MapState commit 5506be87dc3b8f4c4425a259cf255a168fd7b127 Author: Kenneth Knowles Date: 2017-06-22T20:51:46Z This closes #3420: DataflowRunner: Reject SetState and MapState commit 497cfabea7d6dcee0c5d327022678c571c3ec487 Author: Kenneth Knowles Date: 2017-06-22T18:31:28Z Add window matcher for pane info commit d4e5db51a025a831ddf4e3bc0e003caebabf647b Author: Kenneth Knowles Date: 2017-06-22T18:56:53Z Tidy LateDataDroppingDoFnRunner commit 1c1f239501349f5120b0d619c4eea9c435500b78 Author: Kenneth Knowles Date: 2017-06-22T19:52:42Z ReduceFnTester can advance clocks without firing timers commit 795760d370bcbe28e1f0ca373ad4c8c841e6e6b5 Author: Kenneth Knowles Date: 2017-06-22T19:53:15Z ReduceFnTester assertion for windows that have data buffered commit 412fd7eab9e58a4d412f4dff5ffec023610b4f22 Author: Kenneth Knowles Date: 2017-06-22T19:56:14Z Drop late data in ReduceFnTester commit 50c43d96adb8c2523cf38c09f32e241eacc47823 Author: Kenneth Knowles Date: 2017-06-22T19:56:34Z Do not GC windows based on processing time timer! commit fda589c00c8920e76cfc9aaa87cecfa94077599d Author: Kenneth Knowles Date: 2017-06-22T20:04:23Z Add test reproducing BEAM-2505, ignored commit d2b384a20dbb0213d0f63e74713a06d63bad8d39 Author: Kenneth Knowles Date: 2017-06-22T20:05:42Z Add tests for corner cases of processing time timers commit 7645c44b9bae9a77192fbe6744918cfb95ca1a3a Author: Kenneth Knowles Date: 2017-06-22T22:41:02Z This closes #3425: Fix processing time timer handling in ReduceFnRunner Add tests for corner cases of processing time timers Add test reproducing BEAM-2505, ignored Do not GC windows based on processing time timer! Drop late data in ReduceFnTester ReduceFnTester assertion for windows that have data buffered ReduceFnTester can advance clocks without firing timers Tidy LateDataDroppingDoFnRunner Add window matcher for pane info commit 5d6ad19958d0a2394f9e33720a04cc954279a7e7 Author: Robert Bradshaw Date: 2017-06-22T19:44:23Z Remove fn api bundle descriptor translation. commit a882e8f3a33c4a430f55d53b65285123c5a4f50d Author: Robert Bradshaw Date: 2017-06-22T19:46:13Z Remove unused (and untested) initial splittling logic. commit 7a9f762e440168d48e3b3b2930dd83885402e51a Author: Robert Bradshaw Date: 2017-06-23T00:05:32Z Closes #3424 commit 799173fac4e07dab4547ba21971922336cd72c62 Author: Eugene Kirpichov Date: 2017-06-21T22:21:32Z Uses KV in SplittableParDo expansion instead of ElementAndRestriction This is a workaround for the following issue. ElementAndRestriction is in runners-core, which may be shaded by runners (and is shaded by Dataflow runner), hence it should be *both* produced and consumed by workers - but currently it's produced by (shaded) SplittableParDo and consumed by (differently shaded) ProcessFn in the runner's worker code. There are several ways out of this, e.g. moving EAR into the SDK (icky because it's an implementation detail of SplittableParDo), or using a type that's already in the SDK. There may be other more complicated ways too. commit 6f12e7d3d6a9fbd0d7bc1a6136542bd503cb0f2b Author: Eugene Kirpichov Date: 2017-06-22T21:59:36Z Bump Dataflow worker to 0622 commit 8860cceb710f28931fe5ec792b0db09277f0d2a9 Author: Eugene Kirpichov Date: 2017-06-23T00:56:44Z This closes #3417: [BEAM-1377] Uses KV in SplittableParDo expansion instead of ElementAndRestriction commit cab4d8969e7f95b0ece59838ad2d578e75d38823 Author: Kenneth Knowles Date: 2017-06-22T03:25:31Z DataflowRunner: Reject merging windowing for stateful ParDo commit 5f8cfa7412e8f4ce97d6df4413fecbea22b4b5e1 Author: Kenneth Knowles Date: 2017-06-23T03:05:28Z This closes #3419: DataflowRunner: Reject merging windowing for stateful ParDo commit 649994b353afe28c917969609c7a1a47a4f39aaf Author: Rune Fevang Date: 2017-06-15T11:51:12Z Allow output from FinishBundle in DoFnTester commit b3c36256e32f4a4a616d0e152686d29f1174c5cd Author: Kenneth Knowles Date: 2017-06-23T03:07:48Z This closes #3367: Allow output from FinishBundle in DoFnTester commit 8dcda6e40355af13f4d92fcd44aae4539a225a4a Author: Sourabh Bajaj Date: 2017-06-23T00:08:20Z [BEAM-2497] Fix the reading of concat gzip files commit 336b7f1cffc8709cd480a243a16de95bce39b53e Author: Chamikara Jayalath Date: 2017-06-23T05:13:05Z This closes #3428 commit f291713b28e3ba0246a8c0a710c71506cd0a0f91 Author: Etienne Chauchot Date: 2017-06-21T08:39:39Z [BEAM-2489] Use dynamic ES port in HIFIOWithElasticTest ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastructure@apache.org or file a JIRA ticket with INFRA. ---