Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates
 209.85.215.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPcDmSvedhpumixnWC_+kndrnQBGwKeieZVFzMXqh2gqhjoxtQ@mail.gmail.com>
References: 
 <CAPcDmSvedhpumixnWC_+kndrnQBGwKeieZVFzMXqh2gqhjoxtQ@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 28 Nov 2011 15:10:20 -0800
Message-ID: 
 <CADY20s6PQUiBv2QnsbSS+9TPSv8BBru2v76eigqeA0FSPJAY+Q@mail.gmail.com>
Subject: Re: unit tests
To: dev@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Haven't had a chance to review the full note below, but one request:
can we please put this in the code tree itself in a file like
TESTING.txt or README.testing.txt or something?

It'd be nice to be able to refer back to it from within the code tree.

-Todd

On Mon, Nov 28, 2011 at 12:29 PM, N Keywal <nkeywal@gmail.com> wrote:
> Hi,
>
> The surefire modification seems to work well, so we can move to the final
> step.
> Please find below the last proposition for the test. The behavior describ=
ed
> can be activated with:
> =A0- a commit of hbase-4847
> =A0- a modification of the jenkins config to run mvn verify vs. mvn tests=
.
>
> The main points are:
> =A0- should we run small & medium tests only by default on a developer
> machine =3D> here answer is yes
> =A0- should we run all tests on the central build =3D> here answer is yes
> =A0- should we stop when a small test fails =3D> here answer is yes
> =A0- should we stop when a medium test fails =3D> here answer is yes
>
>
> Also, I just had a discussion with Jesse, who think we should keep/have t=
he
> notion of integration test. If retained, the points above must be impacte=
d
> to take this into account.
>
>
> fyi, there are today:
> =A0- 416 small tests, executed in ~3 minutes
> =A0- 489 medium tests, executed in ~35 minutes (without parallelization)
> =A0- 280 large tests, executed in ~90 minutes (without parallelization)
>
> --
> 1) Running tests
> HBase tests are divided in three categories: small, medium and large, wit=
h
> corresponding JUnit categories: SmallTests, MediumTests, LargeTests.
>
> - Small tests are executed in a shared JVM. We put in this category all t=
he
> tests that can be executed quickly (the maximum execution time for a test
> is 15 seconds, and they do not use a cluster) in a shared jvm.
> - Medium tests represents tests that must be executed before proposing a
> patch. They are designed to run in less than 30 minutes altogether, and a=
re
> quite stable in their results. They're designed to last less than 50
> seconds individually. They can use a cluster, and each of them is execute=
d
> in a separate JVM.
> - Large tests are everything else. They are typically integration tests,
> regression tests for specific bugs, timeout tests, performance tests. Som=
e
> of them can be flaky. They are executed before a commit on the
> pre-integration machines. They can be run on the developer machine as wel=
l.
>
> Commands are:
> 1) mvn test
> =A0- execute small tests in a single JVM and medium tests in a separate J=
VM
> for each test
> =A0- medium tests are NOT executed if there is an error in a small test
> =A0- large tests are NOT executed
> =A0- there is one report for small tests, and one report for medium tests=
 -if
> they are executed-
>
> 2) mvn verify
> =A0- execute small tests in a single JVM then medium tests in a separate =
JVM
> for each test, then large tests in a separate JVM as well.
> =A0- medium tests are NOT executed if there is an error in a small test
> =A0- large tests are NOT executed if there is an error in a small or medi=
um
> test
> =A0- there is one report by test category, small, medium and large
>
> 3) mvn test -P localTests -Dtest=3DmyTests
> =A0- remove any category effect (without this specific profile, the profi=
les
> are taken into account)
> =A0- use actually the official release of surefire & the old connector to
> junit
> =A0- tests are executed in separated JVM
> - You will see a new message at the end of the report: "[INFO] Tests are
> skipped". It's harmless.
>
> 4) mvn test -P runAllTests
> =A0- execute small tests in a single JVM then medium & large tests in a
> separate JVM for each test
> =A0- medium and large tests are NOT executed if there is an error in a sm=
all
> test
> =A0- large tests are NOT executed if there is an error in a small or medi=
um
> test
> =A0- there are one report for small tests, and one report for medium & la=
rge
> tests -if they are executed-
>
> 5) Various other profiles
> =A0- mvn test -P runSmallTests =A0 - execute small tests only, =A0in a si=
ngle JVM.
> =A0- mvn test -P runMediumTests =A0 - execute medium tests in a single JV=
M.
> =A0- mvn test -P runLargeTests =A0 - execute medium tests in a single JVM=
.
>
> It's as well possible to use the script 'hbasetests.sh'. This script runs
> the medium and large tests in parallel with two maven instances, and
> provide a single report. It must be executed from the directory which
> contains the pom.xml. Commands are:
> ./dev-support/hbasetests.sh =A0 =A0 =A0 =A0 =A0 =A0 =A0- execute small an=
d medium tests
> ./dev-support/hbasetests.sh runAllTests =A0- execute all tests
> ./dev-support/hbasetests.sh replayFailed - rerun the failed tests a secon=
d
> time, in a separate jvm and without parallelisation.
>
> 2) Writing tests
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests must be written to support parallel execution on the same
> machine, hence should not use shared resources as fixed ports or fixed fi=
le
> names.
> - Tests should not overlog. More than 100 lines/second makes the logs
> complex to read and use i/o that are hence not available for the other
> tests.
> - Tests can be written with HBaseTestingUtility . This class offers helpe=
r
> functions to create a temp directory and do the cleanup, or to start a
> cluster.
>
> - Categories and execution time
> =A0- All tests must be categorized, if not they could be skipped.
> =A0- All tests should be written to be as fast as possible.
> =A0- Small tests should last less than 15 seconds, and must not have any
> side effect.
> =A0- Medium tests should last less than 45 seconds.
> =A0- large tests should last less than 3 minutes, this ensure a good
> parallelization for the ones using it, and ease the analysis when the tes=
t
> fails.
>
> - Sleeps:
> =A0 =A0- Whenever possible, tests should not use sleep, but rather waitin=
g for
> the real event. This is faster and clearer for the reader.
> =A0 =A0- Tests should not do a 'Thread.sleep' without testing an ending
> condition. This allows understanding what the test is waiting for.
> Moreover, the test will work whatever the machine performances.
> =A0 =A0- Sleep should be minimal to be as fast as possible. Waiting for a
> variable should be done in a 40ms sleep loop. Waiting for a socket
> operation should be done in a 200 ms sleep loop.
>
> - Tests using =A0a cluster:
> =A0 =A0- Tests using a HRegion do not have to start a cluster: A region c=
an
> use the local file system.
> =A0 =A0- Start/stopping a cluster cost around 10 seconds. They should not=
 be
> started per test method but per test class.
> =A0 =A0- Started cluster must be shutdown using
> HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> =A0 =A0- As most as possible, tests should use the default settings for t=
he
> cluster. When they don't, they should document it. This will allow to sha=
re
> the cluster later.
>


--=20
Todd Lipcon
Software Engineer, Cloudera