hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Meil <doug.m...@explorysmedical.com>
Subject Re: Speeding up tests
Date Tue, 04 Oct 2011 00:33:41 GMT
Thanks Jesse.  Great write up!




On 10/3/11 4:55 PM, "Jesse Yates" <jesse.k.yates@gmail.com> wrote:

>Hey everyone,
>
>There has been a bunch of work recently on speeding up the testing to make
>it easier for developers to iterate quickly on new features fixes. Part of
>the problem is that the test suite takes anywhere from 1-2 hrs to run and
>have some apparently non-deterministic hanging of tests.
>
>TL;DR To speed it all up, the attack plan would be:
>(1) Move long running tests to be integration tests,
>(2) Use a build server for patches so people only run unit tests locally,
>(3) We add unit tests when integration tests are breaking a lot but unit
>tests pass,
>(4) Go from forked to single jvm unit tests,
>(5) Add in surefire parallelization
>(6) Entertain using HBaseTestingUtilFactory
>
>I recently chatted with Stack and Doug about ways around this. Here is
>what
>we came up with:
>
>(1) Break up long running tests from medium and short (2 mins max) tests
>and
>move the former to be IntegrationTests.  This was based on Todd's
>suggestion
>in HBASE-4438. Either naming them integration tests or long running
>functional tests, they would become part of the 'mvn verify' rather than
>the
>'mvn test' suite of tests. Starting point would be to  use Doug's
>spreadsheet from HBASE-4448 and when its done, the script from HBASE-4480.
>
>Right now, that means a LOT of tests are going to shift, but it means when
>developers run 'mvn test' the amount of time spent running unit tests will
>be cut down dramatically (hopefully towards the sub 10 -15 mins range)
>
>There is an implicit problem here: if the soon to be integration tests
>capture functionality that is not covered by unit tests, then people may
>incorrectly think that they are not breaking things. Therefore, we would
>do
>(2) and (3):
>
>(2) Add a patch continuous integration server that goes and actually
>builds
>and tests patches as they come in. This would run 'mvn verify' and ensure
>that the patch actually isn't breaking high level/complex functionality.
>It
>would be a requirement before patches are committed that they pass this
>build.
>
>(3) If we find that the unit tests aren't covering a certain level of
>functionality that is constantly breaking on the build server, we add more
>unit tests of the breaking functionality to ensure the unit tests are more
>complete and provide more assurances to developers when running them.
>
>This would be an ongoing process of comparing the integration tests vs.
>the
>unit tests.
>
>(4) Once we have a true unit test suite, we should be able to go from
>'forked' jvm mode back to a single jvm for running tests. Unit tests
>should
>not do crazy fault injection, full failure scenarios, so they should be
>able
>to cleanup after themselves. This means we are going to get some speedup
>from not spinning up a new jvm for each test.
>
>(5) Once we are running in non-forked mode, we can try turning on
>parallelized test execution in surefire, which does parallel runs in a
>single jvm.
>
>(6) Once things all run in a single jvm, using the HBaseTestUtilFactory
>(HBASE-4448) make sense reuse the mini clusters across tests.
>
>What does everyone think of this approach?
>
>Thanks!
>
>-Jesse Yates


Mime
View raw message