hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Bautin (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
Date Sun, 06 Nov 2011 01:40:51 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144884#comment-13144884
] 

Mikhail Bautin commented on HBASE-4737:
---------------------------------------

I tried to run trunk unit tests using my map-reduce test runner (which unfortunately only
works in Facebook environment at the moment). To my surprise, it worked extremely well without
HBASE-4746, confirming N's observation. The job completed in under 10 minutes, just like the
one for 89-fb, and ran 239 test classes with 1096 tests. The only failure was TestHRegionInfo.testGetSetOfHTD
(an NPE; will investigate). I started working on HBASE-4767 patch for trunk, but most of its
value will be in refactoring and cleanup, since it looks like there are very few bugs left
in the trunk related to not passing configuration from MiniHBaseCluster to clients in unit
tests.
                
> Split the tests in small/medium/large; allow small tests to be ran in parallel within
a single JVM
> --------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-4737
>                 URL: https://issues.apache.org/jira/browse/HBASE-4737
>             Project: HBase
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.94.0
>            Reporter: nkeywal
>            Assignee: nkeywal
>            Priority: Minor
>         Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch,
20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch,
hbasetests.sh
>
>
> 1) Split the tests in 3 categories
>  - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
>  - medium: <45s, no flaky, useful to detect bugs immediatly
>  - large: remaining
>  
> 2) Allow to run a subset: developpers should need to run only small and medium before
submitting a patch
>  - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit
category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in
small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step
(and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer	3	31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker	8	4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster	1	3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor	5	30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin	1	0.762
> org.apache.hadoop.hbase.client.TestHCM	3	21.961
> org.apache.hadoop.hbase.client.TestHTablePool	18	26.274
> org.apache.hadoop.hbase.client.TestHTableUtil	2	16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD	3	24.629
> org.apache.hadoop.hbase.client.TestMetaScanner	1	16.365
> org.apache.hadoop.hbase.client.TestMultiParallel	10	34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter	3	27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol	44	16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading	5	31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint	2	32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort	1	13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove	1	16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver	3	29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass	2	14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface	5	33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort	1	16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove	1
18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver	3	19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter	1	19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache	5	24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache	7	19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock	7	25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache	7	0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv	8	40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager	2	0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException	1	1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner	1	2.953
> org.apache.hadoop.hbase.master.TestMaster	1	18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler	2	20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager	10	13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion	3	21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler	3	0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead	4	1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter	3	22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed	3	29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion	57	28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager	1	0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner	6	19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics	1	2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations	6	3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol	6	20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker	5	2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary	1	16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort	1	22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay	5	29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink	5	23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager	1	0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer	1	25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin	1	16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable	3	17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter	3	17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource	3	17.314
> org.apache.hadoop.hbase.rest.TestRowResource	11	17.73
> org.apache.hadoop.hbase.rest.TestScannerResource	5	26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters	10	20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource	2	22.441
> org.apache.hadoop.hbase.rest.TestStatusResource	2	14.884
> org.apache.hadoop.hbase.rest.TestTableResource	8	25.658
> org.apache.hadoop.hbase.rest.TestTransform	1	15.966
> org.apache.hadoop.hbase.rest.TestVersionResource	7	19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize	1	18.154
> org.apache.hadoop.hbase.TestInfoServers	2	14.9
> org.apache.hadoop.hbase.TestMultiVersions	3	27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer	1	20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge	1	0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager	2	0.215
> org.apache.hadoop.hbase.util.TestFSUtils	2	19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck	6	41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge	1	0.054
> org.apache.hadoop.hbase.util.TestMergeTable	1	27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter	4	28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable	1	0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker	2	0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster	1	84.463
> org.apache.hadoop.hbase.client.TestAdmin	33	369.833
> org.apache.hadoop.hbase.client.TestFromClientSide	49	154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps	8	66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout	4	47.309
> org.apache.hadoop.hbase.client.TestShell	1	69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat	6	24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce	1	63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat	8	152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery	3	74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles	6	35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan	11	516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce	1	92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed	1	42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting	4	112.606
> org.apache.hadoop.hbase.master.TestMasterFailover	4	76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable	1	20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions	3	22.043
> org.apache.hadoop.hbase.master.TestRestartCluster	2	36.325
> org.apache.hadoop.hbase.master.TestRollingRestart	1	54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction	1	40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad	1	47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster	4	43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit	28	230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog	9	59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling	3	310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication	2	75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication	1	47.873
> org.apache.hadoop.hbase.replication.TestReplication	7	163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction	1	57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility	7	73.654
> org.apache.hadoop.hbase.TestRegionRebalancing	1	58.25
> org.apache.hadoop.hbase.TestZooKeeper	7	86.267
> org.apache.hadoop.hbase.util.TestMergeTool	1	257.223
> SMALL
> Everyting else

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message