hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nkeywal (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4712) Document rules for writing tests
Date Wed, 30 Nov 2011 18:27:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160211#comment-13160211
] 

nkeywal commented on HBASE-4712:
--------------------------------

Updated version. I suppose there is a tool to put that in the existing xml. I naively thought
that open office would do it, but it's not the case...

--
1) Running tests
HBase tests are divided in three categories: small, medium and large, with corresponding JUnit
categories: SmallTests, MediumTests, LargeTests.

Small tests are executed in a shared JVM. We put in this category all the tests that can be
executed quickly (the maximum execution time for a test is 15 seconds, and they do not use
a cluster) in a shared jvm.
Medium tests represents tests that must be executed before proposing a patch. They are designed
to run in less than 30 minutes altogether, and are quite stable in their results. They're
designed to last less than 50 seconds individually. They can use a cluster, and each of them
is executed in a separate JVM.
Large tests are everything else. They are typically integration tests, regression tests for
specific bugs, timeout tests, performance tests. Some of them can be flaky. They are executed
before a commit on the pre-integration machines. They can be run on the developer machine
as well.

Commands are:
1.1) mvn test
- execute small tests in a single JVM and medium tests in a separate JVM for each test
- medium tests are NOT executed if there is an error in a small test
- large tests are NOT executed
- there is one report for small tests, and one report for medium tests if they are executed

1.2) mvn test -P runAllTests
- execute small tests in a single JVM then medium and large tests in a separate JVM for each
test.
- medium and large tests are NOT executed if there is an error in a small test
- there is one report for small tests, and one report for medium and large tests if they are
executed

1.3) mvn test -P localTests -Dtest=myTests
- remove any category effect (without this specific profile, the profiles are taken into account)
- use actually the official release of surefire & the old connector to junit
- tests are executed in separated JVM
- you will see a new message at the end of the report: "[INFO] Tests are skipped". It's harmless.

1.4) mvn test -P runAllTests
- execute small tests in a single JVM then medium & large tests in a separate JVM for
each test
- medium and large tests are NOT executed if there is an error in a small test
- large tests are NOT executed if there is an error in a small or medium test
- there are one report for small tests, and one report for medium & large tests if they
are executed

1.5) Various other profiles
mvn test -P runSmallTests - execute small tests only, in a single JVM.
mvn test -P runMediumTests - execute medium tests in a single JVM.
mvn test -P runLargeTests - execute medium tests in a single JVM.

It's as well possible to use the script 'hbasetests.sh'. This script runs the medium and large
tests in parallel with two maven instances, and provide a single report. It must be executed
from the directory which contains the pom.xml. Commands are:
./dev-support/hbasetests.sh - execute small and medium tests
./dev-support/hbasetests.sh runAllTests - execute all tests
./dev-support/hbasetests.sh replayFailed - rerun the failed tests a second time, in a separate
jvm and without parallelisation.

2) Writing tests
Tests rules & hints are:
- As most as possible, tests should be written as small tests.
- All tests must be written to support parallel execution on the same machine, hence should
not use shared resources as fixed ports or fixed file names.
- Tests should not overlog. More than 100 lines/second makes the logs complex to read and
use i/o that are hence not available for the other tests.
- Tests can be written with HBaseTestingUtility . This class offers helper functions to create
a temp directory and do the cleanup, or to start a cluster.

Categories and execution time
- All tests must be categorized, if not they could be skipped.
- All tests should be written to be as fast as possible.
- Small tests should last less than 15 seconds, and must not have any side effect.
- Medium tests should last less than 45 seconds.
- large tests should last less than 3 minutes, this ensure a good parallelization for people
using it, and ease the analysis when the test fails.

Sleeps
- Whenever possible, tests should not use Thread.sleep, but rather waiting for the real event
they need. This is faster and clearer for the reader.
- Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding
what the test is waiting for. Moreover, the test will work whatever the machine performances.
- Sleep should be minimal to be as fast as possible. Waiting for a variable should be done
in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.

Tests using a cluster
- Tests using a HRegion do not have to start a cluster: A region can use the local file system.
- Start/stopping a cluster cost around 10 seconds. They should not be started per test method
but per test class.
- Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans
the directories.
- As most as possible, tests should use the default settings for the cluster. When they don't,
they should document it. This will allow to share the cluster later.
                
> Document rules for writing tests
> --------------------------------
>
>                 Key: HBASE-4712
>                 URL: https://issues.apache.org/jira/browse/HBASE-4712
>             Project: HBase
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.92.0
>            Reporter: nkeywal
>            Assignee: nkeywal
>            Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding
JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds.
They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They
can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression
tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence
should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read
and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to
create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
>     - Tests should not do a 'Thread.sleep' without testing an ending condition. This
allows understanding what the test is waiting for. Moreover, the test will work whatever the
machine performances.
>     - Sleep should be minimal to be as fast as possible. Waiting for a variable should
be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep
loop.
> - Tests using cluster:
>     - Tests using a HRegion do not have to start a cluster: A region can use the local
file system.
>     - Start/stopping a cluster cost around 10 seconds. They should not be started per
test method but per class.
>     - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster,
which cleans the directories.
>     - As most as possible, tests should use the default settings for the cluster. When
they don't, they should document it. This will allow to share the cluster later.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message