beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-160) Port 'NexMark Queries' to Beam for use as integration test
Date Wed, 30 Mar 2016 20:40:25 GMT

    [ https://issues.apache.org/jira/browse/BEAM-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218812#comment-15218812
] 

ASF GitHub Bot commented on BEAM-160:
-------------------------------------

GitHub user mshields822 opened a pull request:

    https://github.com/apache/incubator-beam/pull/99

    [BEAM-160] Import NexMark integration tests.

    BUILDS ON https://github.com/apache/incubator-beam/pull/85
    
    This is an initial import of the 'NexMark' integration tests.
    All Google Dataflow specific code has been isolated.
    It could do with more cleanup, but wanted to get it out in the sunlight. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mshields822/incubator-beam nexmark

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/99.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #99
    
----
commit dbc39a605bf09f2d1f1cae05de4caaf176abe6c1
Author: Mark Shields <markshields@google.com>
Date:   2016-03-28T20:30:37Z

    Initial import

commit 4fa0b3d98c302c8b710106610b5aca0721436f87
Author: Mark Shields <markshields@google.com>
Date:   2016-03-28T20:37:23Z

    Initial import II

commit 4ba8f304821a09af12e4fd8703d64c16b1349256
Author: Mark Shields <markshields@google.com>
Date:   2016-03-28T21:58:49Z

    Formatting busywork

commit e78d9a8b220874b87107312a6d13670e798184f7
Author: Mark Shields <markshields@google.com>
Date:   2016-03-30T18:52:34Z

    pub/sub source/sink and grpc client refactor
    
    Move some pubsub source/sink machinery into
    PubSubGrpcClient so that it's api can be grpc
    and protoc neutral.

commit a7291da3b27a657b8cd2deeda87f158f97240b76
Author: Mark Shields <markshields@google.com>
Date:   2016-03-30T19:22:20Z

    Unused imports

commit c2e8219050c904ba7a8cbb57199f95b61e88bec7
Author: Mark Shields <markshields@google.com>
Date:   2016-03-30T19:39:21Z

    Import order

commit 444937c53d219c10f2705e41e5c3268abccd6c0e
Author: Mark Shields <markshields@google.com>
Date:   2016-03-28T23:25:29Z

    Copy over. Break deps on RetryHttpInitializer

commit 30585ad3529b0799043d3cef9d2824337b631baa
Author: Mark Shields <markshields@google.com>
Date:   2016-03-30T20:04:10Z

    Unused import

commit c5d663ed79c6a89b05c580f8340024a211482f9e
Author: Mark Shields <markshields@google.com>
Date:   2016-03-30T20:24:57Z

    Limit dataflow dependencies

----


> Port 'NexMark Queries' to Beam for use as integration test
> ----------------------------------------------------------
>
>                 Key: BEAM-160
>                 URL: https://issues.apache.org/jira/browse/BEAM-160
>             Project: Beam
>          Issue Type: Test
>          Components: testing
>            Reporter: Mark Shields
>            Assignee: Mark Shields
>
> A while back we implemented the 'queries' from
>   http://datalab.cs.pdx.edu/niagara/NEXMark/
> as Gooogle Dataflow pipelines. We found them useful
> for uncovering performance problems with the sdk, our runners,
> and our service. Many of those problems only manifested under
> high load, multi-day runs, or with high 'backlog' on the incoming
> pub/sub subscriptions.
> We thus think they would be useful for other runners.
> Disclaimer: Though the original 'queries' were proposed as a way to
> benchmark 'continuous SQL' implementations, we have so far only
> used them for internal A/B and regression testing and have not validated
> them as representative of customer workloads. We would thus discourage their use for
competitive benchmarks without more work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message