hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From N Keywal <nkey...@gmail.com>
Subject Re: fastest way to run tests
Date Tue, 28 Aug 2012 05:35:40 GMT
Yes, jps will show the processes when you start the medium & large tests
(see 15.5.2. Unit Tests in hbase ref guide)

Hadoop-qa builds are executed with 4 processes. If locally you have
"Address already in use" or ""Filesystem closed" it's a bug or a regression.
We will need to increase the number of process from time to time to keep a
reasonable test execution time while adding new tests.

Rules are (section 15 again):

   - As much as possible, tests should be written as category small tests.
   - All tests must be written to support parallel execution on the same
   machine, hence they should not use shared resources as fixed ports or fixed
   file names.
   - Tests should not overlog. More than 100 lines/second makes the logs
   complex to read and use i/o that are hence not available for the other
   tests.
   - Tests can be written with HBaseTestingUtility. This class offers
   helper functions to create a temp directory and do the cleanup, or to start
   a cluster. Categories and execution time
   - All tests must be categorized, if not they could be skipped.
   - All tests should be written to be as fast as possible.
   - Small category tests should last less than 15 seconds, and must not
   have any side effect.
   - Medium category tests should last less than 50 seconds.
   - Large category tests should last less than 3 minutes. This should
   ensure a good parallelization for people using it, and ease the analysis
   when the test fails.

Cheers,

Nicolas



On Tue, Aug 28, 2012 at 5:06 AM, Jimmy Xiang <jxiang@cloudera.com> wrote:

> How can I verify that the thread count setting is used?
>
> Will jps show that many surefire processes?
>
> Thanks,
> Jimmy
>
> On Mon, Aug 27, 2012 at 5:23 PM, Matt Corgan <mcorgan@hotpads.com> wrote:
> > Thanks  - that helps a good bit.  I do get some failures from things like
> > "Filesystem closed" and "Address already in use", but sounds like that's
> > not unexpected right now.
> >
> >
> > On Mon, Aug 27, 2012 at 2:17 PM, Stack <stack@duboce.net> wrote:
> >
> >> On Mon, Aug 27, 2012 at 2:16 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >> > Thanks N for sharing the knowledge.
> >> >
> >> > Looks like some test needs to use random port(s):
> >> >
> >> >
> >> testStopDuringStart(org.apache.hadoop.hbase.master.TestMasterNoCluster):
> >> > Problem binding to sea-lab-0/10.249.196.101:60000 : Address already
> in
> >> use
> >> >   testFailover(org.apache.hadoop.hbase.master.TestMasterNoCluster):
> >> Problem
> >> > binding to sea-lab-0/10.249.196.101:60000 : Address already in use
> >> >
> testCatalogDeploys(org.apache.hadoop.hbase.master.TestMasterNoCluster):
> >> > Problem binding to sea-lab-0/10.249.196.101:60000 : Address already
> in
> >> use
> >> >
> >> > Cheers
> >> >
> >>
> >> File an issue Ted?
> >> St.Ack
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message