hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From N Keywal <nkey...@gmail.com>
Subject unit tests - pom.xml & surefire changed - categories are available
Date Sat, 19 Nov 2011 20:36:04 GMT
Hi,

The pom.xml using the new surefire & junit has been committed by Ted today.
We're using a specific repo setup by Gary, as we had to deploy private
versions of Surefire and JUnit, for reasons detailed in HBASE-4763.

It means it's now possible to use categories. I updated the doc in
HBASE-4712 on how to use them, I'm adding the last version of this doc at
the end of this mail.
We may have specific bugs, as this feature is new (and relies on a specific
component of surefire that we were not using before). Contact me if you
have any issue.

The tests are split in two separate executions, one for the tests within a
single JVM, one for the others. By default there all executed in different
JVM, and the second report is skipped. So you will see a new message at the
end of the report: "[INFO] Tests are skipped". It's harmless.

If everything runs fine, I will propose to change the default to execute
only the dev tests. As well, the small tests are not yet executed in
parallel, still this "baby steps" approach :-).

Regards,

N.

PS 1: Don't forget to categorize all your new tests!
PS 2: for hbasetests.sh, it needs hbase-4826 for the new pom.xml


---- Tests documentation
1) Running tests
HBase tests are divided in three categories: small, medium and large, with
corresponding JUnit categories: SmallTests, MediumTests, LargeTests.

- Small tests are executed in a shared JVM. We put in this category all the
tests that can be executed quickly (the maximum execution time for a test
is 15 seconds and they do not use a cluster) in a shared jvm.
- Medium tests represents tests that must be executed before proposing a
patch. They are designed to run in less than 30 minutes altogether, and are
quite stable in their results. They're designed to less than 50 seconds.
They can use a cluster, and each of them are executed in a separate JVM.
- Large tests are everything else. They are typically integration tests,
regression tests for specific bugs, timeout tests, performance tests. Some
of them can be flaky. They are executed before a commit on the
pre-integration machines. They can be run on the developper as well.

Commands are:
 - mvn test               - execute all tests in a separate JVM. All
results are presented in a single report.
 - mvn test -P runDevTests     - execute small tests in a single JVM and
medium tests in separates jvm. There are two reports.
 - mvn test -P runAllTests     - execute small tests in a single JVM and
medium and large tests in separates jvm. There are two reports, one for
small, one for medium and large.
 - mvn test -P runSmallTests   - execute small tests in a single JVM.

It's as well possible to use the script 'hbasetests.sh'. This script runs
the medium and large tests in parallel with two maven instances, and
provide a single report. It must be executed from the directory which
contains the pom.xml. Commands are:
./dev-support/hbasetests.sh              - execute small and medium tests
./dev-support/hbasetests.sh runAllTests  - execute all tests
./dev-support/hbasetests.sh replayFailed - rerun the failed tests a second
time, in a separate jvm and without parallelisation.

2) Writing tests
Tests rules & hints are:
- As most as possible, tests should be written as small tests.
- All tests must be written to support parallel execution on the same
machine, hence should not use shared resources as fixed ports or fixed file
names.
- Tests should not overlog. More than 100 lines/second makes the logs
complex to read and use i/o that are hence not available for the other
tests.
- Tests can be written with HBaseTestingUtility . This class offers helper
functions to create a temp directory and do the cleanup, or to start a
cluster.

- Categories and executin time
  - All tests must be categorized, if not they could be skipped.
  - All tests should be written to be as fast as possible.
  - Small tests should last less than 15 seconds, and must not have any
side effect.
  - Medium tests should last less than 45 seconds.
  - large tests should last less than 3 minutes, this ensure a good
parallelisation for the ones using it, and ease the analysis when the test
fails.

- Sleeps:
    - Whenever possible, tests should not use sleep, but rather waiting for
the real event. This is faster and clearer for the reader.
    - Tests should not do a 'Thread.sleep' without testing an ending
condition. This allows understanding what the test is waiting for.
Moreover, the test will work whatever the machine performances.
    - Sleep should be minimal to be as fast as possible. Waiting for a
variable should be done in a 40ms sleep loop. Waiting for a socket
operation should be done in a 200 ms sleep loop.

- Tests using cluster:
    - Tests using a HRegion do not have to start a cluster: A region can
use the local file system.
    - Start/stopping a cluster cost around 10 seconds. They should not be
started per test method but per class.
    - Started cluster must be shutdown using
HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
    - As most as possible, tests should use the default settings for the
cluster. When they don't, they should document it. This will allow to share
the cluster later.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message