hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesse Yates <jesse.k.ya...@gmail.com>
Subject Re: Speeding up tests
Date Tue, 04 Oct 2011 00:30:38 GMT
Yeah, they would still run in forked mode. There are also a lot of cases
where we are testing edge failure scenarios and injecting errors into the
cluster that really need to be in their own jvm.

We are trying to cut down on everything that we can to speed up the build.
Agree that a lot of the time is on spin up/down of mini-clusters, so if we
can just use one across multiple tests, we can see a big speed up. When we
move them into single jvm mode there is definitely going to be some issues,
but lets cross that bridge when we come to it :)

-Jesse Yates

On Mon, Oct 3, 2011 at 5:25 PM, Jonathan Hsieh <jon@cloudera.com> wrote:

> I'm assuming that the longrunning/integration tests will still run as new
> jvm instances?
>
> I've been working on hbase offline recovery tools whose tests (in current
> incarnation) requires hbase cluster shutdown and restart.   I assume these
> would be integration tests.  I've also found at least in these cases, it is
> mini cluster spinup and spindown that is more costly than jvm spin up.
>  I've
> also found that here is some file handle leakage in the mini cluster
> utility
> classes and other possbible leakage due to statics and what such as
> HConnectionManager which may make using a single jvm infeasible for the
> long
> running tests until that gets fixed.
>
> Jon.
>
> On Mon, Oct 3, 2011 at 1:55 PM, Jesse Yates <jesse.k.yates@gmail.com>
> wrote:
>
> > Hey everyone,
> >
> > There has been a bunch of work recently on speeding up the testing to
> make
> > it easier for developers to iterate quickly on new features fixes. Part
> of
> > the problem is that the test suite takes anywhere from 1-2 hrs to run and
> > have some apparently non-deterministic hanging of tests.
> >
> > TL;DR To speed it all up, the attack plan would be:
> > (1) Move long running tests to be integration tests,
> > (2) Use a build server for patches so people only run unit tests locally,
> > (3) We add unit tests when integration tests are breaking a lot but unit
> > tests pass,
> > (4) Go from forked to single jvm unit tests,
> > (5) Add in surefire parallelization
> > (6) Entertain using HBaseTestingUtilFactory
> >
> > I recently chatted with Stack and Doug about ways around this. Here is
> what
> > we came up with:
> >
> > (1) Break up long running tests from medium and short (2 mins max) tests
> > and
> > move the former to be IntegrationTests.  This was based on Todd's
> > suggestion
> > in HBASE-4438. Either naming them integration tests or long running
> > functional tests, they would become part of the 'mvn verify' rather than
> > the
> > 'mvn test' suite of tests. Starting point would be to  use Doug's
> > spreadsheet from HBASE-4448 and when its done, the script from
> HBASE-4480.
> >
> > Right now, that means a LOT of tests are going to shift, but it means
> when
> > developers run 'mvn test' the amount of time spent running unit tests
> will
> > be cut down dramatically (hopefully towards the sub 10 -15 mins range)
> >
> > There is an implicit problem here: if the soon to be integration tests
> > capture functionality that is not covered by unit tests, then people may
> > incorrectly think that they are not breaking things. Therefore, we would
> do
> > (2) and (3):
> >
> > (2) Add a patch continuous integration server that goes and actually
> builds
> > and tests patches as they come in. This would run 'mvn verify' and ensure
> > that the patch actually isn't breaking high level/complex functionality.
> It
> > would be a requirement before patches are committed that they pass this
> > build.
> >
> > (3) If we find that the unit tests aren't covering a certain level of
> > functionality that is constantly breaking on the build server, we add
> more
> > unit tests of the breaking functionality to ensure the unit tests are
> more
> > complete and provide more assurances to developers when running them.
> >
> > This would be an ongoing process of comparing the integration tests vs.
> the
> > unit tests.
> >
> > (4) Once we have a true unit test suite, we should be able to go from
> > 'forked' jvm mode back to a single jvm for running tests. Unit tests
> should
> > not do crazy fault injection, full failure scenarios, so they should be
> > able
> > to cleanup after themselves. This means we are going to get some speedup
> > from not spinning up a new jvm for each test.
> >
> > (5) Once we are running in non-forked mode, we can try turning on
> > parallelized test execution in surefire, which does parallel runs in a
> > single jvm.
> >
> > (6) Once things all run in a single jvm, using the HBaseTestUtilFactory
> > (HBASE-4448) make sense reuse the mini clusters across tests.
> >
> > What does everyone think of this approach?
> >
> > Thanks!
> >
> > -Jesse Yates
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message