Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B6CCE200CDD for ; Mon, 24 Jul 2017 07:20:59 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id B51EC1648A6; Mon, 24 Jul 2017 05:20:59 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 04E0A1648A2 for ; Mon, 24 Jul 2017 07:20:58 +0200 (CEST) Received: (qmail 43423 invoked by uid 500); 24 Jul 2017 05:20:58 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 43414 invoked by uid 99); 24 Jul 2017 05:20:58 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Jul 2017 05:20:58 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id C29F2E0896; Mon, 24 Jul 2017 05:20:57 +0000 (UTC) From: bastiliu To: commits@beam.apache.org Reply-To: commits@beam.apache.org References: In-Reply-To: Subject: [GitHub] beam pull request #3624: [BEAM-1899] Implementation of JStorm runner Content-Type: text/plain Message-Id: <20170724052057.C29F2E0896@git1-us-west.apache.org> Date: Mon, 24 Jul 2017 05:20:57 +0000 (UTC) archived-at: Mon, 24 Jul 2017 05:20:59 -0000 GitHub user bastiliu reopened a pull request: https://github.com/apache/beam/pull/3624 [BEAM-1899] Implementation of JStorm runner Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- @kennknowles @peihe Could you please help to review the implementation of JStorm runner? Known issues: - There are still two test suites which were failed(CountingSourceTest, GenerateSequenceTest). I am looking on them. - SetState, SplittableParDo, AttemptedMetric, CommittedMetrics, and TestStream were not supported yet. So these parts are excluded in the integration test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bastiliu/beam jstorm-runner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3624.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3624 ---- commit 731555c1174aa4aaea70baf25e001e8bccd16142 Author: basti.lj Date: 2017-05-02T03:12:58Z Refactor: move JStorm runner code to Beam repo. commit ef81cd8e43cbc9379fee3a40a0c50bb8c267f9f9 Author: basti.lj Date: 2017-05-12T08:24:00Z Support startBundle & finishBundle on jstorm batch mode commit 0a5b7e69c6edc289052b44b9c9efd10d4ec1d5a8 Author: basti.lj Date: 2017-05-12T08:25:43Z Update to internal version 0.7.0-jstorm commit 4d0a594df51102578c86935a6db7bad853894ae1 Author: Pei He Date: 2017-05-17T08:28:16Z Upgrade Beam to 2.0.0. commit ec91b660e15e782016e2ec9518247c99921be31e Author: Pei He Date: 2017-05-18T03:27:20Z Internal: Upgrade fixup for imports. commit 9d2ddb4553fb7347eb3c5900dee03a02d85cd925 Author: Pei He Date: 2017-05-18T03:30:36Z jstorm-runner: 2.0.0 upgrade fixups. commit c12c768e9f29b73d005adc367259bc11c8bdabfb Author: Pei He Date: 2017-05-18T04:46:22Z Internal: Upgrade fixup for states. commit 9bd97cf35e0e1285fc6459708a5d062905befe74 Author: Pei He Date: 2017-05-18T05:03:40Z Internal: Upgrade fixup for removing aggregators. commit ff316f61132fd55914f7d19f320d244f4053832e Author: Pei He Date: 2017-05-18T05:15:23Z Internal: Upgrade fixup for others. commit a09d761e3171ee62eda4718255144db9107d88cb Author: Pei He Date: 2017-05-18T12:16:55Z Internal: Implement Beam Metrics. commit 1d85413c49335d9427bcc01c900bc500fdb7c5d4 Author: Pei He Date: 2017-05-18T12:22:07Z Internal: workaround before calling finishBundle is fixed. commit fdcd3e9fc4ed561d164f791c848c99c0f59742eb Author: basti.lj Date: 2017-05-19T09:24:38Z Fix that bug that state of timers were not persisted correctly. commit 49968800bb731d1f34322615ddf35cc1e0be6f8c Author: basti.lj Date: 2017-05-19T15:13:33Z fix bug that startBundle/finishBundle was not called after fire timers by updating watermark commit 715512e92049fc2f018b56c044c2beb11fadebc5 Author: basti.lj Date: 2017-05-22T02:23:18Z Revert "Internal: workaround before calling finishBundle is fixed." This reverts commit c0a40cf4317a7fa63b401c7f5fecea6b17355b55. commit d9cde0537ad1b80ca7455e5cc3af0332e208350b Author: basti.lj Date: 2017-05-23T04:32:28Z Map local TupleTag to external tag before output in DoFn commit 3ab3e73fd10408c5ee212c179a1d8bc147a8dcbd Author: basti.lj Date: 2017-05-23T04:39:13Z Add combine test cases commit e9775e4d10d6fd573496ba018dde0f4d68c6bd1d Author: Pei He Date: 2017-05-25T09:54:52Z Add WordCountBenchmark for JStorm runner. commit 0b2b26df9060e2a961621c770e2fe07751c6acce Author: Pei He Date: 2017-05-26T08:12:03Z fixup: remove print messages. commit 10dfe21738a2f2bd1c198c03b521b583d8af0a2b Author: Pei He Date: 2017-05-27T09:47:27Z Benchmark: add StateInternalsBenchmark. commit fa9c498f30a690d5e64dcc48b28f58e15012f4b4 Author: basti.lj Date: 2017-06-02T02:39:04Z 1. Fix incorrect asserted value in stateInternalsTest 2. Deactivate source reader when closing spout 3. Fix typo in pom.xml commit 343421d8ccf0a8ed4e63c64c816756400c706510 Author: basti.lj Date: 2017-06-06T08:17:36Z 1. Fix bug of state internal test 2. Improve performance of bag state by reducing duplicated wrtie/read ops commit 7ee5ef4127ab849645bd6c136efe1c987fda3aa5 Author: basti.lj Date: 2017-06-06T09:27:52Z Fix incorrect asserted result of CombineTest commit ca79d9fc728a2ebf3f899d7c36b88945386ad954 Author: basti.lj Date: 2017-06-08T08:45:27Z support Iterable in JStormBagState.read commit 294d7933c4bbb2ec7ebf6242ef3793ce12a22938 Author: basti.lj Date: 2017-06-08T08:47:08Z Build immutable list for "Iterable" type output value commit 94e83eff35cc9cc7a6cacda225cc3725d2317ee4 Author: basti.lj Date: 2017-06-09T07:47:07Z Support keys, values and entries interface of MapState commit e4aa7c692fc14dfc67f57f924b82ad1f022bc134 Author: basti.lj Date: 2017-06-12T12:13:58Z Replace immutable list with array list because "null" is a valid value which is not permitted in immutable list. commit 6c161da1c50e47362eb7c4dbd5ca0730bdc3cab3 Author: basti.lj Date: 2017-06-14T06:34:07Z Add unit test for CoGroupByKey and adding massive data into BagState commit 65d11974d15aca5293ff77c2169e6de798216635 Author: basti.lj Date: 2017-06-26T03:10:12Z Support user to define parallelism number of composite PTransform commit 1d5c3a8bca3c772b0cb88c50b0721f48e3660201 Author: basti.lj Date: 2017-06-26T03:21:56Z Add missing file for last commit of "parallelism number support on composite transform" commit 2912db32ed4a7d5b0875fdf9eb351618c5384cee Author: basti.lj Date: 2017-07-04T07:14:16Z 1. Fix the bug that different DoFn executors use same state instance 2. Support stateful keyed DoFn ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastructure@apache.org or file a JIRA ticket with INFRA. ---