hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: Speeding up tests
Date Tue, 04 Oct 2011 00:25:59 GMT
I'm assuming that the longrunning/integration tests will still run as new
jvm instances?

I've been working on hbase offline recovery tools whose tests (in current
incarnation) requires hbase cluster shutdown and restart.   I assume these
would be integration tests.  I've also found at least in these cases, it is
mini cluster spinup and spindown that is more costly than jvm spin up.  I've
also found that here is some file handle leakage in the mini cluster utility
classes and other possbible leakage due to statics and what such as
HConnectionManager which may make using a single jvm infeasible for the long
running tests until that gets fixed.


On Mon, Oct 3, 2011 at 1:55 PM, Jesse Yates <jesse.k.yates@gmail.com> wrote:

> Hey everyone,
> There has been a bunch of work recently on speeding up the testing to make
> it easier for developers to iterate quickly on new features fixes. Part of
> the problem is that the test suite takes anywhere from 1-2 hrs to run and
> have some apparently non-deterministic hanging of tests.
> TL;DR To speed it all up, the attack plan would be:
> (1) Move long running tests to be integration tests,
> (2) Use a build server for patches so people only run unit tests locally,
> (3) We add unit tests when integration tests are breaking a lot but unit
> tests pass,
> (4) Go from forked to single jvm unit tests,
> (5) Add in surefire parallelization
> (6) Entertain using HBaseTestingUtilFactory
> I recently chatted with Stack and Doug about ways around this. Here is what
> we came up with:
> (1) Break up long running tests from medium and short (2 mins max) tests
> and
> move the former to be IntegrationTests.  This was based on Todd's
> suggestion
> in HBASE-4438. Either naming them integration tests or long running
> functional tests, they would become part of the 'mvn verify' rather than
> the
> 'mvn test' suite of tests. Starting point would be to  use Doug's
> spreadsheet from HBASE-4448 and when its done, the script from HBASE-4480.
> Right now, that means a LOT of tests are going to shift, but it means when
> developers run 'mvn test' the amount of time spent running unit tests will
> be cut down dramatically (hopefully towards the sub 10 -15 mins range)
> There is an implicit problem here: if the soon to be integration tests
> capture functionality that is not covered by unit tests, then people may
> incorrectly think that they are not breaking things. Therefore, we would do
> (2) and (3):
> (2) Add a patch continuous integration server that goes and actually builds
> and tests patches as they come in. This would run 'mvn verify' and ensure
> that the patch actually isn't breaking high level/complex functionality. It
> would be a requirement before patches are committed that they pass this
> build.
> (3) If we find that the unit tests aren't covering a certain level of
> functionality that is constantly breaking on the build server, we add more
> unit tests of the breaking functionality to ensure the unit tests are more
> complete and provide more assurances to developers when running them.
> This would be an ongoing process of comparing the integration tests vs. the
> unit tests.
> (4) Once we have a true unit test suite, we should be able to go from
> 'forked' jvm mode back to a single jvm for running tests. Unit tests should
> not do crazy fault injection, full failure scenarios, so they should be
> able
> to cleanup after themselves. This means we are going to get some speedup
> from not spinning up a new jvm for each test.
> (5) Once we are running in non-forked mode, we can try turning on
> parallelized test execution in surefire, which does parallel runs in a
> single jvm.
> (6) Once things all run in a single jvm, using the HBaseTestUtilFactory
> (HBASE-4448) make sense reuse the mini clusters across tests.
> What does everyone think of this approach?
> Thanks!
> -Jesse Yates

// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message