flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li, Chengxiang" <chengxiang...@intel.com>
Subject RE: [Proposal] Create a separate sub module for benchmark test
Date Wed, 23 Sep 2015 02:11:28 GMT
Thanks for the correctness, Stephan.
For the micro benchmark, I think it should be better to make a new maven project inside Flink.


-----Original Message-----
From: ewenstephan@gmail.com [mailto:ewenstephan@gmail.com] On Behalf Of Stephan Ewen
Sent: Tuesday, September 22, 2015 8:14 PM
To: dev@flink.apache.org
Subject: Re: [Proposal] Create a separate sub module for benchmark test

Sounds like a nice idea!

Do you want to make this a new maven project as part of the Flink repository, or create a
dedicated repository for that?

BTW: We are currently not mixing microbenchmarks with test execution. The code for these benchmarks
resides in the test scope of the projects (so it is not packaged), but it is not executed
as part of the UnitTests or IntegrationTests.


On Tue, Sep 22, 2015 at 12:22 PM, Li, Chengxiang <chengxiang.li@intel.com>

> Hi, folks
> During work on Flink, I found several micro benchmarks which come from 
> different modules, these benchmarks measure on manual, annotated with 
> Junit annotations, so they got executed during unit test as well. 
> There are some shortage on current implementation:
> 1.       Benchmark test performance instead of feature, and normally, it
> takes much more time than unit test. Mixed benchmark with unit test 
> would expand the CI check time.
> 2.       With mixed with other tests, no warm up, no standalone process...
> these benchmarks result may not very accurate.
> Although looks easy, there are actually many pitfalls about benchmark, 
> so I suggest we create a new sub module for all benchmark test, and 
> import JMH(
> http://openjdk.java.net/projects/code-tools/jmh/) as the benchmark 
> framework. With the help of JMH, we should get:
> 1.       More available metrics.
> 2.       More accurate result.
> 3.       Focus on benchmark logic only, no need to worry about measure
> logic.
> Thanks
> Chengxiang
View raw message