beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chamikara Jayalath (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-3215) Add a performance test for HBaseIO
Date Mon, 20 Nov 2017 15:52:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259396#comment-16259396
] 

Chamikara Jayalath commented on BEAM-3215:
------------------------------------------

Ah, thanks for pointing out that issue. I also created https://issues.apache.org/jira/browse/BEAM-3214
which is a duplicate of https://issues.apache.org/jira/browse/BEAM-1783.

Both performance and integration tests can indeed share the same pipeline and datastore setup.
The idea is that we can create two versions of the pipeline that conforms to following requirements.


(1) 
Can be run in a single VM
Runs a single-node data store instance
The dataset should be small (only up to few 1000s of rows/elements).

(2)
Executes multiple worker in the runner.
Runs a multi-node datastore instance.
The dataset should be large (at least 10 GBs of data).

We can run version (1) in existing post-commit Jenkins test suites while version (2) should
be run as a separate Jenkins test suite that runs less regularly.

BTW are you hoping to add integration tests for HBaseIO ? If so that's great. Otherwise we'll
continue and add you as a reviewer.

> Add a performance test for HBaseIO
> ----------------------------------
>
>                 Key: BEAM-3215
>                 URL: https://issues.apache.org/jira/browse/BEAM-3215
>             Project: Beam
>          Issue Type: Test
>          Components: sdk-java-extensions
>            Reporter: Chamikara Jayalath
>            Assignee: Kamil Szewczyk
>
> We should add a large scale performance test for HBaseIO. We could use PerfKitBenchmarker
based performance testing framework [1] to manage a Kubernetes based muti-node HBase cluster
and to publish benchmark results.
> Example docker image to use: https://hub.docker.com/r/dajobe/hbase/
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message