beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bastiliu <...@git.apache.org>
Subject [GitHub] beam pull request #3624: [BEAM-1899] Implementation of JStorm runner
Date Mon, 24 Jul 2017 05:20:57 GMT
GitHub user bastiliu reopened a pull request:

    https://github.com/apache/beam/pull/3624

    [BEAM-1899] Implementation of JStorm runner

    Follow this checklist to help us incorporate your contribution quickly and easily:
    
     - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/)
filed for the change (usually before you start working on it).  Trivial changes like typos
do not require a JIRA issue.  Your pull request should address just this issue, without pulling
in other changes.
     - [ ] Each commit in the pull request should have a meaningful subject line and body.
     - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`,
where you replace `BEAM-XXX` with the appropriate JIRA issue.
     - [ ] Write a pull request description that is detailed enough to understand what the
pull request does, how, and why.
     - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
     - [ ] If this contribution is large, please file an Apache [Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.pdf).
    
    ---
    
    @kennknowles  @peihe  Could you please help to review the implementation of JStorm runner?

    
    Known issues:
    - There are still two test suites which were failed(CountingSourceTest, GenerateSequenceTest).
I am looking on them.
    - SetState, SplittableParDo, AttemptedMetric, CommittedMetrics, and TestStream were not
supported yet. So these parts are excluded in the integration test.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/bastiliu/beam jstorm-runner

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3624
    
----
commit 731555c1174aa4aaea70baf25e001e8bccd16142
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-05-02T03:12:58Z

    Refactor: move JStorm runner code to Beam repo.

commit ef81cd8e43cbc9379fee3a40a0c50bb8c267f9f9
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-05-12T08:24:00Z

    Support startBundle & finishBundle on jstorm batch mode

commit 0a5b7e69c6edc289052b44b9c9efd10d4ec1d5a8
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-05-12T08:25:43Z

    Update to internal version 0.7.0-jstorm

commit 4d0a594df51102578c86935a6db7bad853894ae1
Author: Pei He <hepei.hp@alibaba-inc.com>
Date:   2017-05-17T08:28:16Z

    Upgrade Beam to 2.0.0.

commit ec91b660e15e782016e2ec9518247c99921be31e
Author: Pei He <hepei.hp@alibaba-inc.com>
Date:   2017-05-18T03:27:20Z

    Internal: Upgrade fixup for imports.

commit 9d2ddb4553fb7347eb3c5900dee03a02d85cd925
Author: Pei He <hepei.hp@alibaba-inc.com>
Date:   2017-05-18T03:30:36Z

    jstorm-runner: 2.0.0 upgrade fixups.

commit c12c768e9f29b73d005adc367259bc11c8bdabfb
Author: Pei He <hepei.hp@alibaba-inc.com>
Date:   2017-05-18T04:46:22Z

    Internal: Upgrade fixup for states.

commit 9bd97cf35e0e1285fc6459708a5d062905befe74
Author: Pei He <hepei.hp@alibaba-inc.com>
Date:   2017-05-18T05:03:40Z

    Internal: Upgrade fixup for removing aggregators.

commit ff316f61132fd55914f7d19f320d244f4053832e
Author: Pei He <hepei.hp@alibaba-inc.com>
Date:   2017-05-18T05:15:23Z

    Internal: Upgrade fixup for others.

commit a09d761e3171ee62eda4718255144db9107d88cb
Author: Pei He <hepei.hp@alibaba-inc.com>
Date:   2017-05-18T12:16:55Z

    Internal: Implement Beam Metrics.

commit 1d85413c49335d9427bcc01c900bc500fdb7c5d4
Author: Pei He <hepei.hp@alibaba-inc.com>
Date:   2017-05-18T12:22:07Z

    Internal: workaround before calling finishBundle is fixed.

commit fdcd3e9fc4ed561d164f791c848c99c0f59742eb
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-05-19T09:24:38Z

    Fix that bug that state of timers were not persisted correctly.

commit 49968800bb731d1f34322615ddf35cc1e0be6f8c
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-05-19T15:13:33Z

    fix bug that startBundle/finishBundle was not called after fire timers by updating watermark

commit 715512e92049fc2f018b56c044c2beb11fadebc5
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-05-22T02:23:18Z

    Revert "Internal: workaround before calling finishBundle is fixed."
    
    This reverts commit c0a40cf4317a7fa63b401c7f5fecea6b17355b55.

commit d9cde0537ad1b80ca7455e5cc3af0332e208350b
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-05-23T04:32:28Z

    Map local TupleTag to external tag before output in DoFn

commit 3ab3e73fd10408c5ee212c179a1d8bc147a8dcbd
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-05-23T04:39:13Z

    Add combine test cases

commit e9775e4d10d6fd573496ba018dde0f4d68c6bd1d
Author: Pei He <hepei.hp@alibaba-inc.com>
Date:   2017-05-25T09:54:52Z

    Add WordCountBenchmark for JStorm runner.

commit 0b2b26df9060e2a961621c770e2fe07751c6acce
Author: Pei He <hepei.hp@alibaba-inc.com>
Date:   2017-05-26T08:12:03Z

    fixup: remove print messages.

commit 10dfe21738a2f2bd1c198c03b521b583d8af0a2b
Author: Pei He <pei@apache.org>
Date:   2017-05-27T09:47:27Z

    Benchmark: add StateInternalsBenchmark.

commit fa9c498f30a690d5e64dcc48b28f58e15012f4b4
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-06-02T02:39:04Z

    1. Fix incorrect asserted value in stateInternalsTest
    2. Deactivate source reader when closing spout
    3. Fix typo in pom.xml

commit 343421d8ccf0a8ed4e63c64c816756400c706510
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-06-06T08:17:36Z

    1. Fix bug of state internal test
    2. Improve performance of bag state by reducing duplicated wrtie/read ops

commit 7ee5ef4127ab849645bd6c136efe1c987fda3aa5
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-06-06T09:27:52Z

    Fix incorrect asserted result of CombineTest

commit ca79d9fc728a2ebf3f899d7c36b88945386ad954
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-06-08T08:45:27Z

    support Iterable in JStormBagState.read

commit 294d7933c4bbb2ec7ebf6242ef3793ce12a22938
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-06-08T08:47:08Z

    Build immutable list for "Iterable" type output value

commit 94e83eff35cc9cc7a6cacda225cc3725d2317ee4
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-06-09T07:47:07Z

    Support keys, values and entries interface of MapState

commit e4aa7c692fc14dfc67f57f924b82ad1f022bc134
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-06-12T12:13:58Z

    Replace immutable list with array list because "null" is a valid value which is not permitted
in immutable list.

commit 6c161da1c50e47362eb7c4dbd5ca0730bdc3cab3
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-06-14T06:34:07Z

    Add unit test for CoGroupByKey and adding massive data into BagState

commit 65d11974d15aca5293ff77c2169e6de798216635
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-06-26T03:10:12Z

    Support user to define parallelism number of composite PTransform

commit 1d5c3a8bca3c772b0cb88c50b0721f48e3660201
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-06-26T03:21:56Z

    Add missing file for last commit of "parallelism number support on composite transform"

commit 2912db32ed4a7d5b0875fdf9eb351618c5384cee
Author: basti.lj <basti.lj@alibaba-inc.com>
Date:   2017-07-04T07:14:16Z

    1. Fix the bug that different DoFn executors use same state instance
    2. Support stateful keyed DoFn

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message