hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nkeywal (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4965) Monitor the open file descriptors and the threads counters during the unit tests
Date Wed, 07 Dec 2011 14:36:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164427#comment-13164427 ] 

nkeywal commented on HBASE-4965:
--------------------------------

Here are the possible leaks. I am gonna fix some of them in a separate patch. Leaks on "SmallTests" are critical, because we the JVM is used for multiple tests.

This one should be studied:
client.TestAdmin#testCheckHBaseAvailableClosesConnection: 523 threads (was 298), 913 file descriptors (was 488).  -thread leak?-  -file handle leak?- 

As the limit on hadoop-QA is 1024 open file descriptor, it's not far from hitting this limit. Especially is another test is ran after this one.


avro.TestAvroServer#testTableAdminAndMetadata: 140 threads (was 130), 255 file descriptors (was 253).  -thread leak?-  -file handle leak?- 
avro.TestAvroServer#testFamilyAdminAndMetadata: 144 threads (was 140), 255 file descriptors (was 255).  -thread leak?- 
avro.TestAvroServer#testDML: 146 threads (was 144), 255 file descriptors (was 255).  -thread leak?- 
catalog.TestCatalogTrackerOnCluster#testBadOriginalRootLocation: 23 threads (was 4), 127 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
catalog.TestCatalogTracker#testThatIfMETAMovesWeAreNotified: 9 threads (was 8), 84 file descriptors (was 79).  -thread leak?-  -file handle leak?- 
catalog.TestCatalogTracker#testInterruptWaitOnMetaAndRoot: 10 threads (was 9), 86 file descriptors (was 84).  -file handle leak?- 
catalog.TestCatalogTracker#testVerifyRootRegionLocationFails: 11 threads (was 9), 89 file descriptors (was 85).  -thread leak?-  -file handle leak?- 
catalog.TestMetaReaderEditorNoCluster#testRideOverServerNotRunning: 7 threads (was 4), 85 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
catalog.TestMetaReaderEditor#testGetRegionsCatalogTables: 190 threads (was 185), 360 file descriptors (was 354).  -thread leak?-  -file handle leak?- 
catalog.TestMetaReaderEditor#testTableExists: 191 threads (was 187), 365 file descriptors (was 360).  -thread leak?-  -file handle leak?- 
catalog.TestMetaReaderEditor#testGetRegion: 193 threads (was 191), 370 file descriptors (was 365).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testDeleteEditUnknownColumnFamilyAndOrTable: 254 threads (was 246), 423 file descriptors (was 417).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testDisableAndEnableTable: 273 threads (was 254), 452 file descriptors (was 423).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testDisableAndEnableTables: 294 threads (was 272), 482 file descriptors (was 452).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testCreateTable: 294 threads (was 294), 491 file descriptors (was 482).  -file handle leak?- 
client.TestAdmin#testOnlineChangeTableSchema: 295 threads (was 294), 494 file descriptors (was 491).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testCreateTableWithRegions: 296 threads (was 294), 490 file descriptors (was 490).  -thread leak?- 
client.TestAdmin#testTableExist: 297 threads (was 296), 494 file descriptors (was 490).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testForceSplit: 303 threads (was 297), 487 file descriptors (was 494).  -thread leak?- 
client.TestAdmin#testForceSplitMultiFamily: 309 threads (was 293), 499 file descriptors (was 464).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testEnableDisableAddColumnDeleteColumn: 312 threads (was 309), 505 file descriptors (was 499).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testCreateBadTables: 313 threads (was 312), 507 file descriptors (was 505).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testCreateTableRPCTimeOut: 312 threads (was 313), 526 file descriptors (was 507).  -file handle leak?- 
client.TestAdmin#testReadOnlyTable: 314 threads (was 312), 530 file descriptors (was 526).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testCloseRegionThatFetchesTheHRIFromMeta: 315 threads (was 312), 513 file descriptors (was 507).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testGetTableRegions: 309 threads (was 308), 512 file descriptors (was 499).  -thread leak?-  -file handle leak?- 
client.TestAdmin#testCheckHBaseAvailableClosesConnection: 523 threads (was 298), 913 file descriptors (was 488).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testKeepDeletedCells: 261 threads (was 246), 437 file descriptors (was 414).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testRegionCacheDeSerialization: 276 threads (was 261), 485 file descriptors (was 437).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testRegionCachePreWarm: 277 threads (was 276), 488 file descriptors (was 485).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testWeirdCacheBehaviour: 285 threads (was 277), 500 file descriptors (was 488).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testSuperSimple: 284 threads (was 276), 466 file descriptors (was 443).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testMaxKeyValueSize: 285 threads (was 284), 466 file descriptors (was 466).  -thread leak?- 
client.TestFromClientSide#testFilters: 290 threads (was 285), 477 file descriptors (was 466).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testKeyOnlyFilter: 291 threads (was 290), 477 file descriptors (was 477).  -thread leak?- 
client.TestFromClientSide#testSimpleMissing: 292 threads (was 291), 477 file descriptors (was 477).  -thread leak?- 
client.TestFromClientSide#testSingleRowMultipleFamily: 294 threads (was 292), 500 file descriptors (was 477).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testNull: 298 threads (was 294), 505 file descriptors (was 500).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testVersions: 301 threads (was 298), 523 file descriptors (was 505).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testVersionLimits: 302 threads (was 301), 523 file descriptors (was 523).  -thread leak?- 
client.TestFromClientSide#testDeletes: 303 threads (was 302), 523 file descriptors (was 523).  -thread leak?- 
client.TestFromClientSide#testJiraTest867: 304 threads (was 303), 532 file descriptors (was 523).  -file handle leak?- 
client.TestFromClientSide#testJiraTest861: 304 threads (was 303), 540 file descriptors (was 532).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testJiraTest33: 307 threads (was 304), 547 file descriptors (was 540).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testJiraTest1014: 307 threads (was 307), 550 file descriptors (was 547).  -file handle leak?- 
client.TestFromClientSide#testJiraTest1182: 309 threads (was 307), 557 file descriptors (was 550).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testJiraTest52: 313 threads (was 309), 565 file descriptors (was 557).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testDuplicateVersions: 314 threads (was 313), 579 file descriptors (was 565).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testUpdates: 315 threads (was 314), 580 file descriptors (was 579).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testMajorCompactionBetweenTwoUpdates: 303 threads (was 305), 564 file descriptors (was 561).  -file handle leak?- 
client.TestFromClientSide#testGet_EmptyTable: 310 threads (was 303), 578 file descriptors (was 564).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testPut: 313 threads (was 310), 582 file descriptors (was 578).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testRowsPutBufferedOneFlush: 313 threads (was 312), 580 file descriptors (was 580).  -thread leak?- 
client.TestFromClientSide#testHBase737: 312 threads (was 313), 592 file descriptors (was 580).  -file handle leak?- 
client.TestFromClientSide#testListTables: 313 threads (was 312), 604 file descriptors (was 592).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testMiscHTableStuff: 316 threads (was 313), 608 file descriptors (was 604).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testGetClosestRowBefore: 316 threads (was 316), 611 file descriptors (was 608).  -file handle leak?- 
client.TestFromClientSide#testPoolBehavior: 317 threads (was 315), 611 file descriptors (was 611).  -thread leak?- 
client.TestFromClientSide#testClientPoolRoundRobin: 317 threads (was 317), 614 file descriptors (was 611).  -file handle leak?- 
client.TestFromClientSide#testClientPoolThreadLocal: 318 threads (was 317), 614 file descriptors (was 614).  -thread leak?- 
client.TestFromClientSide#testScanMetrics: 320 threads (was 319), 640 file descriptors (was 614).  -thread leak?-  -file handle leak?- 
client.TestFromClientSide#testCacheOnWriteEvictOnClose: 321 threads (was 320), 641 file descriptors (was 640).  -thread leak?-  -file handle leak?- 
client.TestHCM#testRegionCaching: 143 threads (was 130), 265 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
client.TestHCM#testConnectionUniqueness: 180 threads (was 142), 358 file descriptors (was 262).  -thread leak?-  -file handle leak?- 
client.TestHCM#testClosing: 186 threads (was 180), 376 file descriptors (was 358).  -thread leak?-  -file handle leak?- 
client.TestHCM#testCreateConnection: 190 threads (was 186), 386 file descriptors (was 376).  -thread leak?-  -file handle leak?- 
client.TestHTableUtil#testBucketPut: 135 threads (was 130), 255 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
client.TestHTableUtil#testBucketBatch: 138 threads (was 135), 255 file descriptors (was 255).  -thread leak?- 
client.TestInstantSchemaChangeFailover#testInstantSchemaChangeWhileRSCrash: 27 threads (was 4), 197 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChangeFailover#testInstantSchemaChangeWhileRandomRSCrashAndStart: 35 threads (was 27), 241 file descriptors (was 197).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChangeFailover#testInstantSchemaChangeWhileMasterFailover: 37 threads (was 35), 251 file descriptors (was 241).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChange#testInstantSchemaChangeForModifyTable: 34 threads (was 4), 230 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChange#testInstantSchemaChangeForAddColumn: 47 threads (was 34), 280 file descriptors (was 230).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChange#testInstantSchemaChangeForModifyColumn: 55 threads (was 47), 294 file descriptors (was 280).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChange#testInstantSchemaChangeForDeleteColumn: 64 threads (was 55), 334 file descriptors (was 294).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChange#testInstantSchemaChangeWhenTableIsNotEnabled: 71 threads (was 64), 387 file descriptors (was 334).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChange#testConcurrentInstantSchemaChangeForAddColumn: 72 threads (was 71), 424 file descriptors (was 387).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChange#testConcurrentInstantSchemaChangeAndLoadBalancerRun: 93 threads (was 72), 497 file descriptors (was 424).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChange#testLoadBalancerBlocksDuringSchemaChangeRequests: 97 threads (was 93), 512 file descriptors (was 497).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChange#testInstantSchemaChangeBlocksDuringLoadBalancerRun: 111 threads (was 97), 592 file descriptors (was 512).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChange#testInstantSchemaJanitor: 110 threads (was 111), 606 file descriptors (was 592).  -file handle leak?- 
client.TestInstantSchemaChangeSplit#testInstantSchemaChangeExclusions: 24 threads (was 4), 180 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChangeSplit#testInstantSchemaChangeWhileRSOpenRegionFailure: 40 threads (was 24), 230 file descriptors (was 180).  -thread leak?-  -file handle leak?- 
client.TestInstantSchemaChangeSplit#testConcurrentInstantSchemaChangeAndSplit: 50 threads (was 40), 320 file descriptors (was 230).  -thread leak?-  -file handle leak?- 
client.TestMetaMigrationRemovingHTD#testMetaMigration: 150 threads (was 148), 310 file descriptors (was 310).  -thread leak?- 
client.TestMetaMigrationRemovingHTD#testMasterCrashDuringMetaMigration: 152 threads (was 150), 310 file descriptors (was 310).  -thread leak?- 
client.TestMetaScanner#testMetaScanner: 139 threads (was 130), 255 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
client.TestMultiParallel#testActiveThreadsCount: 210 threads (was 202), 369 file descriptors (was 359).  -thread leak?-  -file handle leak?- 
client.TestMultiParallel#testBatchWithGet: 212 threads (was 210), 369 file descriptors (was 369).  -thread leak?- 
client.TestMultiParallel#testBadFam: 213 threads (was 212), 369 file descriptors (was 369).  -thread leak?- 
client.TestMultiParallel#testFlushCommitsNoAbort: 224 threads (was 162), 388 file descriptors (was 308).  -thread leak?-  -file handle leak?- 
client.TestMultiParallel#testBatchWithPut: 225 threads (was 224), 386 file descriptors (was 388).  -thread leak?- 
client.TestMultiParallel#testBatchWithDelete: 227 threads (was 225), 386 file descriptors (was 386).  -thread leak?- 
client.TestMultiParallel#testHTableDeleteWithList: 229 threads (was 227), 386 file descriptors (was 386).  -thread leak?- 
client.TestMultiParallel#testBatchWithManyColsInOneRowGetAndPut: 229 threads (was 228), 384 file descriptors (was 384).  -thread leak?- 
client.TestMultiParallel#testBatchWithMixedActions: 231 threads (was 229), 384 file descriptors (was 384).  -thread leak?- 
client.TestMultipleTimestamps#testReseeksWithOneColumnMiltipleTimestamp: 136 threads (was 130), 254 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
client.TestMultipleTimestamps#testReseeksWithMultipleColumnOneTimestamp: 140 threads (was 136), 262 file descriptors (was 254).  -thread leak?-  -file handle leak?- 
client.TestMultipleTimestamps#testReseeksWithMultipleColumnMultipleTimestamp: 145 threads (was 140), 264 file descriptors (was 262).  -thread leak?-  -file handle leak?- 
client.TestMultipleTimestamps#testReseeksWithMultipleFiles: 148 threads (was 145), 277 file descriptors (was 264).  -thread leak?-  -file handle leak?- 
client.TestMultipleTimestamps#testWithVersionDeletes: 150 threads (was 148), 281 file descriptors (was 277).  -thread leak?-  -file handle leak?- 
client.TestMultipleTimestamps#testWithMultipleVersionDeletes: 151 threads (was 150), 282 file descriptors (was 281).  -thread leak?-  -file handle leak?- 
client.TestMultipleTimestamps#testWithColumnDeletes: 152 threads (was 151), 283 file descriptors (was 282).  -thread leak?-  -file handle leak?- 
client.TestMultipleTimestamps#testWithFamilyDeletes: 153 threads (was 152), 284 file descriptors (was 283).  -thread leak?-  -file handle leak?- 
client.TestScannerTimeout#test3686a: 161 threads (was 156), 299 file descriptors (was 292).  -thread leak?-  -file handle leak?- 
client.TestScannerTimeout#test3686b: 193 threads (was 161), 325 file descriptors (was 299).  -thread leak?-  -file handle leak?- 
client.TestShell#testRunShellTests: 181 threads (was 130), 266 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
client.TestTimestampsFilter#testTimestampsFilter: 136 threads (was 130), 258 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
client.TestTimestampsFilter#testMultiColumns: 140 threads (was 136), 262 file descriptors (was 258).  -thread leak?-  -file handle leak?- 
client.TestTimestampsFilter#testWithVersionDeletes: 145 threads (was 140), 268 file descriptors (was 262).  -thread leak?-  -file handle leak?- 
coprocessor.TestAggregateProtocol#testRowCountWithValidRange: 212 threads (was 209), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testRowCountAllTable: 215 threads (was 212), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testRowCountWithNullCQ: 218 threads (was 215), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testRowCountWithPrefixFilter: 221 threads (was 218), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testMaxWithValidRange: 224 threads (was 221), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testMaxWithValidRange2: 226 threads (was 224), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testMaxWithValidRangeWithNoCQ: 229 threads (was 226), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testMaxWithValidRange2WithNoCQ: 230 threads (was 229), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testMaxWithFilter: 233 threads (was 230), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testMinWithValidRange: 236 threads (was 233), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testMinWithValidRange2: 238 threads (was 236), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testMinWithValidRangeWithNoCQ: 241 threads (was 238), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testMinWithValidRange2WithNoCQ: 242 threads (was 241), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testMinWithFilter: 245 threads (was 242), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testSumWithValidRange: 248 threads (was 245), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testSumWithValidRange2: 250 threads (was 248), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testSumWithValidRangeWithNoCQ: 253 threads (was 250), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testSumWithValidRange2WithNoCQ: 254 threads (was 253), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testSumWithFilter: 257 threads (was 254), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testAvgWithValidRange: 260 threads (was 257), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testAvgWithValidRange2: 262 threads (was 260), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testAvgWithValidRangeWithNoCQ: 265 threads (was 262), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testAvgWithValidRange2WithNoCQ: 266 threads (was 265), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testAvgWithFilter: 269 threads (was 266), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testStdWithValidRange: 272 threads (was 269), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testStdWithValidRange2: 274 threads (was 272), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testStdWithValidRangeWithNoCQ: 277 threads (was 274), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testStdWithValidRange2WithNoCQ: 278 threads (was 277), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestAggregateProtocol#testStdWithFilter: 281 threads (was 278), 354 file descriptors (was 354).  -thread leak?- 
coprocessor.TestClassLoading#testClassLoadingFromHDFS: 134 threads (was 130), 319 file descriptors (was 253).  -thread leak?-  -file handle leak?- 
coprocessor.TestClassLoading#testClassLoadingFromLocalFS: 136 threads (was 134), 320 file descriptors (was 319).  -thread leak?-  -file handle leak?- 
coprocessor.TestClassLoading#testHBase3810: 140 threads (was 136), 324 file descriptors (was 320).  -thread leak?-  -file handle leak?- 
coprocessor.TestClassLoading#testRegionServerCoprocessorsReported: 146 threads (was 140), 325 file descriptors (was 324).  -thread leak?-  -file handle leak?- 
coprocessor.TestCoprocessorEndpoint#testAggregation: 202 threads (was 198), 335 file descriptors (was 333).  -thread leak?-  -file handle leak?- 
coprocessor.TestMasterCoprocessorExceptionWithRemove#testExceptionFromCoprocessorWhenCreatingTable: 138 threads (was 130), 260 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
coprocessor.TestMasterObserver#testTableOperations: 214 threads (was 190), 367 file descriptors (was 336).  -thread leak?-  -file handle leak?- 
coprocessor.TestMasterObserver#testRegionTransitionOperations: 219 threads (was 214), 372 file descriptors (was 367).  -thread leak?-  -file handle leak?- 
coprocessor.TestRegionObserverBypass#testSimple: 135 threads (was 134), 252 file descriptors (was 252).  -thread leak?- 
coprocessor.TestRegionObserverBypass#testMulti: 136 threads (was 135), 252 file descriptors (was 252).  -thread leak?- 
coprocessor.TestRegionObserverInterface#testRegionObserver: 137 threads (was 130), 258 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
coprocessor.TestRegionObserverInterface#testIncrementHook: 141 threads (was 137), 258 file descriptors (was 258).  -thread leak?- 
coprocessor.TestRegionObserverInterface#testHBase3583: 146 threads (was 141), 258 file descriptors (was 258).  -thread leak?- 
coprocessor.TestRegionObserverInterface#testHBase3758: 148 threads (was 146), 255 file descriptors (was 258).  -thread leak?- 
coprocessor.TestRegionObserverInterface#testCompactionOverride: 151 threads (was 148), 260 file descriptors (was 255).  -thread leak?-  -file handle leak?- 
coprocessor.TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut: 203 threads (was 188), 356 file descriptors (was 332).  -thread leak?-  -file handle leak?- 
coprocessor.TestWALObserver#testWALObserverWriteToWAL: 137 threads (was 130), 257 file descriptors (was 253).  -thread leak?-  -file handle leak?- 
coprocessor.TestWALObserver#testWALCoprocessorReplay: 143 threads (was 137), 273 file descriptors (was 257).  -thread leak?-  -file handle leak?- 
coprocessor.TestWALObserver#testWALObserverLoaded: 145 threads (was 143), 273 file descriptors (was 273).  -thread leak?- 
filter.TestColumnRangeFilter#TestColumnRangeFilterClient: 135 threads (was 130), 261 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
io.hfile.slab.TestSlabCache#testCacheMultiThreaded: 5 threads (was 4), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestCacheOnWrite#testCacheOnWrite[0]: 6 threads (was 4), 75 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
io.hfile.TestFixedFileTrailer#testTrailer[0]: 4 threads (was 4), 78 file descriptors (was 70).  -file handle leak?- 
io.hfile.TestFixedFileTrailer#testTrailer[1]: 4 threads (was 4), 82 file descriptors (was 78).  -file handle leak?- 
io.hfile.TestHFileBlockIndex#testBlockIndex[0]: 4 threads (was 4), 75 file descriptors (was 70).  -file handle leak?- 
io.hfile.TestHFileBlockIndex#testHFileWriterAndReader[0]: 6 threads (was 4), 77 file descriptors (was 75).  -thread leak?-  -file handle leak?- 
io.hfile.TestHFileBlock#testNoCompression: 4 threads (was 4), 71 file descriptors (was 70).  -file handle leak?- 
io.hfile.TestHFileBlock#testReaderV1: 4 threads (was 4), 75 file descriptors (was 71).  -file handle leak?- 
io.hfile.TestHFileBlock#testConcurrentReading: 56 threads (was 4), 75 file descriptors (was 75).  -thread leak?- 
io.hfile.TestLruBlockCache#testBackgroundEvictionThread[0]: 6 threads (was 4), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testCacheSimple[0]: 8 threads (was 6), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testCacheEvictionSimple[0]: 9 threads (was 8), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testCacheEvictionTwoPriorities[0]: 10 threads (was 9), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testCacheEvictionThreePriorities[0]: 11 threads (was 10), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testScanResistance[0]: 12 threads (was 11), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testResizeBlockCache[0]: 13 threads (was 12), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testBackgroundEvictionThread[1]: 15 threads (was 13), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testCacheSimple[1]: 17 threads (was 15), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testCacheEvictionSimple[1]: 18 threads (was 17), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testCacheEvictionTwoPriorities[1]: 19 threads (was 18), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testCacheEvictionThreePriorities[1]: 20 threads (was 19), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testScanResistance[1]: 21 threads (was 20), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestLruBlockCache#testResizeBlockCache[1]: 22 threads (was 21), 70 file descriptors (was 70).  -thread leak?- 
io.hfile.TestReseekTo#testReseekTo: 7 threads (was 7), 94 file descriptors (was 92).  -file handle leak?- 
io.TestHalfStoreFileReader#testHalfScanAndReseek: 7 threads (was 7), 106 file descriptors (was 100).  -file handle leak?- 
ipc.TestDelayedRpc#testDelayedRpcImmediateReturnValue: 18 threads (was 4), 116 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
ipc.TestDelayedRpc#testDelayedRpcDelayedReturnValue: 32 threads (was 18), 155 file descriptors (was 116).  -thread leak?-  -file handle leak?- 
ipc.TestDelayedRpc#testTooManyDelayedRpcs: 46 threads (was 32), 194 file descriptors (was 155).  -thread leak?-  -file handle leak?- 
ipc.TestDelayedRpc#testEndDelayThrowing: 60 threads (was 46), 233 file descriptors (was 194).  -thread leak?-  -file handle leak?- 
mapred.TestTableInputFormat#testTableRecordReader: 135 threads (was 130), 255 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
mapred.TestTableInputFormat#testTableRecordReaderScannerFail: 138 threads (was 135), 255 file descriptors (was 255).  -thread leak?- 
mapred.TestTableInputFormat#testTableRecordReaderScannerTimeout: 141 threads (was 138), 255 file descriptors (was 255).  -thread leak?- 
mapred.TestTableInputFormat#testTableRecordReaderMapreduce: 143 threads (was 141), 255 file descriptors (was 255).  -thread leak?- 
mapred.TestTableInputFormat#testTableRecordReaderScannerFailMapreduce: 144 threads (was 143), 255 file descriptors (was 255).  -thread leak?- 
mapred.TestTableInputFormat#testTableRecordReaderScannerTimeoutMapreduce: 145 threads (was 144), 255 file descriptors (was 255).  -thread leak?- 
mapred.TestTableMapReduce#testMultiRegionTable: 204 threads (was 194), 296 file descriptors (was 288).  -thread leak?-  -file handle leak?- 
mapreduce.TestHFileOutputFormat#test_LATEST_TIMESTAMP_isReplaced: 4 threads (was 4), 74 file descriptors (was 70).  -file handle leak?- 
mapreduce.TestHFileOutputFormat#test_TIMERANGE: 4 threads (was 4), 77 file descriptors (was 74).  -file handle leak?- 
mapreduce.TestHFileOutputFormat#testMRIncrementalLoad: 31 threads (was 4), 189 file descriptors (was 77).  -thread leak?-  -file handle leak?- 
mapreduce.TestHFileOutputFormat#testMRIncrementalLoadWithSplit: 48 threads (was 31), 232 file descriptors (was 189).  -thread leak?-  -file handle leak?- 
mapreduce.TestHFileOutputFormat#testColumnFamilyCompression: 48 threads (was 48), 190 file descriptors (was 186).  -file handle leak?- 
mapreduce.TestHFileOutputFormat#testExcludeMinorCompaction: 64 threads (was 48), 265 file descriptors (was 190).  -thread leak?-  -file handle leak?- 
mapreduce.TestImportTsv#testMROnTable: 15 threads (was 4), 120 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
mapreduce.TestImportTsv#testMROnTableWithCustomMapper: 22 threads (was 15), 141 file descriptors (was 120).  -thread leak?-  -file handle leak?- 
mapreduce.TestLoadIncrementalHFiles#testSimpleLoad: 140 threads (was 130), 262 file descriptors (was 253).  -thread leak?-  -file handle leak?- 
mapreduce.TestLoadIncrementalHFiles#testRegionCrossingLoad: 144 threads (was 140), 282 file descriptors (was 262).  -thread leak?-  -file handle leak?- 
mapreduce.TestLoadIncrementalHFiles#testRegionCrossingRowBloom: 148 threads (was 144), 302 file descriptors (was 282).  -thread leak?-  -file handle leak?- 
mapreduce.TestLoadIncrementalHFiles#testRegionCrossingRowColBloom: 152 threads (was 148), 322 file descriptors (was 302).  -thread leak?-  -file handle leak?- 
mapreduce.TestLoadIncrementalHFiles#testSplitStoreFile: 154 threads (was 152), 329 file descriptors (was 322).  -thread leak?-  -file handle leak?- 
mapreduce.TestLoadIncrementalHFilesSplitRecovery#testBulkLoadPhaseFailure: 135 threads (was 130), 255 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
mapreduce.TestLoadIncrementalHFilesSplitRecovery#testSplitWhileBulkLoadPhase: 178 threads (was 134), 450 file descriptors (was 255).  -thread leak?-  -file handle leak?- 
mapreduce.TestLoadIncrementalHFilesSplitRecovery#testGroupOrSplitPresplit: 213 threads (was 178), 618 file descriptors (was 450).  -thread leak?-  -file handle leak?- 
mapreduce.TestLoadIncrementalHFilesSplitRecovery#testGroupOrSplitFailure: 213 threads (was 213), 622 file descriptors (was 618).  -file handle leak?- 
mapreduce.TestTableInputFormatScan#testScanEmptyToEmpty: 335 threads (was 331), 542 file descriptors (was 485).  -thread leak?-  -file handle leak?- 
mapreduce.TestTableInputFormatScan#testScanEmptyToAPP: 341 threads (was 335), 564 file descriptors (was 542).  -thread leak?-  -file handle leak?- 
mapreduce.TestTableInputFormatScan#testScanEmptyToBBB: 340 threads (was 341), 558 file descriptors (was 544).  -file handle leak?- 
mapreduce.TestTableInputFormatScan#testScanEmptyToOPP: 349 threads (was 340), 549 file descriptors (was 558).  -thread leak?- 
mapreduce.TestTableInputFormatScan#testScanOBBToQPP: 352 threads (was 349), 562 file descriptors (was 549).  -thread leak?-  -file handle leak?- 
mapreduce.TestTableInputFormatScan#testScanYZYToEmpty: 352 threads (was 347), 552 file descriptors (was 549).  -thread leak?-  -file handle leak?- 
mapreduce.TestTableMapReduce#testMultiRegionTable: 204 threads (was 193), 298 file descriptors (was 283).  -thread leak?-  -file handle leak?- 
mapreduce.TestTimeRangeMapRed#testTimeRangeMapRed: 145 threads (was 130), 304 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
master.TestActiveMasterManager#testRestartMaster: 10 threads (was 8), 85 file descriptors (was 79).  -thread leak?-  -file handle leak?- 
master.TestActiveMasterManager#testActiveMasterManagerFromZK: 12 threads (was 10), 90 file descriptors (was 85).  -thread leak?-  -file handle leak?- 
master.TestAssignmentManager#testUnassignWithSplitAtSameTime: 13 threads (was 12), 116 file descriptors (was 112).  -thread leak?-  -file handle leak?- 
master.TestCatalogJanitor#testLastParentCleanedEvenIfDaughterGoneFirst: 11 threads (was 10), 108 file descriptors (was 108).  -thread leak?- 
master.TestDefaultLoadBalancer#testBalanceCluster: 4 threads (was 4), 72 file descriptors (was 70).  -file handle leak?- 
master.TestDistributedLogSplitting#testThreeRSAbort: 45 threads (was 4), 241 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
master.TestDistributedLogSplitting#testOrphanLogCreation: 58 threads (was 45), 292 file descriptors (was 241).  -thread leak?-  -file handle leak?- 
master.TestDistributedLogSplitting#testRecoveredEdits: 80 threads (was 58), 359 file descriptors (was 292).  -thread leak?-  -file handle leak?- 
master.TestHMasterRPCException#testRPCException: 28 threads (was 4), 129 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
master.TestLogsCleaner#testLogCleaning: 12 threads (was 8), 91 file descriptors (was 79).  -thread leak?-  -file handle leak?- 
master.TestMasterFailover#testSimpleMasterFailover: 28 threads (was 4), 158 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
master.TestMasterFailover#testShouldCheckMasterFailOverWhenMETAIsInOpenedState: 40 threads (was 28), 184 file descriptors (was 158).  -thread leak?-  -file handle leak?- 
master.TestMasterFailover#testMasterFailoverWithMockedRIT: 62 threads (was 40), 263 file descriptors (was 184).  -thread leak?-  -file handle leak?- 
master.TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS: 63 threads (was 62), 260 file descriptors (was 263).  -thread leak?- 
master.TestMaster#testMasterOpsWhileSplitting: 141 threads (was 130), 264 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
master.TestMasterRestartAfterDisablingTable#testForCheckingIfEnableAndDisableWorksFineAfterSwitch: 168 threads (was 4), 302 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
master.TestMasterStatusServlet#testStatusTemplateNoTables: 4 threads (was 4), 73 file descriptors (was 72).  -file handle leak?- 
master.TestOpenedRegionHandler#testOpenedRegionHandlerOnMasterRestart: 24 threads (was 4), 130 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
master.TestOpenedRegionHandler#testShouldNotCompeleteOpenedRegionSuccessfullyIfVersionMismatches: 30 threads (was 24), 141 file descriptors (was 130).  -thread leak?-  -file handle leak?- 
master.TestRestartCluster#testRestartClusterAfterKill: 25 threads (was 4), 146 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
master.TestRestartCluster#testClusterRestart: 49 threads (was 25), 224 file descriptors (was 146).  -thread leak?-  -file handle leak?- 
master.TestRollingRestart#testBasicRollingRestart: 312 threads (was 4), 527 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
master.TestSplitLogManager#testTaskCreation: 7 threads (was 4), 80 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
master.TestSplitLogManager#testOrphanTaskAcquisition: 9 threads (was 7), 85 file descriptors (was 80).  -thread leak?-  -file handle leak?- 
master.TestSplitLogManager#testUnassignedOrphan: 12 threads (was 9), 90 file descriptors (was 85).  -thread leak?-  -file handle leak?- 
master.TestSplitLogManager#testMultipleResubmits: 13 threads (was 12), 95 file descriptors (was 90).  -thread leak?-  -file handle leak?- 
master.TestSplitLogManager#testRescanCleanup: 15 threads (was 13), 100 file descriptors (was 95).  -thread leak?-  -file handle leak?- 
master.TestSplitLogManager#testTaskDone: 18 threads (was 15), 105 file descriptors (was 100).  -thread leak?-  -file handle leak?- 
master.TestSplitLogManager#testTaskErr: 21 threads (was 18), 110 file descriptors (was 105).  -thread leak?-  -file handle leak?- 
master.TestSplitLogManager#testTaskResigned: 25 threads (was 21), 111 file descriptors (was 110).  -thread leak?-  -file handle leak?- 
master.TestSplitLogManager#testUnassignedTimeout: 23 threads (was 24), 116 file descriptors (was 111).  -file handle leak?- 
master.TestSplitLogManager#testDeadWorker: 26 threads (was 23), 121 file descriptors (was 116).  -thread leak?-  -file handle leak?- 
master.TestZKBasedOpenCloseRegion#testReOpenRegion: 222 threads (was 219), 382 file descriptors (was 381).  -thread leak?-  -file handle leak?- 
master.TestZKBasedOpenCloseRegion#testRSAlreadyProcessingRegion: 225 threads (was 222), 382 file descriptors (was 382).  -thread leak?- 
master.TestZKBasedOpenCloseRegion#testCloseRegion: 226 threads (was 225), 382 file descriptors (was 382).  -thread leak?- 
regionserver.handler.TestCloseRegionHandler#testFailedFlushAborts: 5 threads (was 4), 76 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
regionserver.handler.TestOpenRegionHandler#testYankingRegionFromUnderIt: 11 threads (was 8), 88 file descriptors (was 79).  -thread leak?-  -file handle leak?- 
regionserver.handler.TestOpenRegionHandler#testFailedOpenRegion: 13 threads (was 11), 93 file descriptors (was 88).  -thread leak?-  -file handle leak?- 
regionserver.handler.TestOpenRegionHandler#testFailedUpdateMeta: 15 threads (was 13), 98 file descriptors (was 93).  -thread leak?-  -file handle leak?- 
regionserver.TestEndToEndSplitTransaction#testMasterOpsWhileSplitting: 139 threads (was 130), 259 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
regionserver.TestFSErrorsExposed#testHFileScannerThrowsErrors: 6 threads (was 4), 75 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
regionserver.TestFSErrorsExposed#testStoreFileScannerThrowsErrors: 6 threads (was 6), 77 file descriptors (was 75).  -file handle leak?- 
regionserver.TestFSErrorsExposed#testFullSystemBubblesFSErrors: 17 threads (was 6), 110 file descriptors (was 77).  -thread leak?-  -file handle leak?- 
regionserver.TestHRegionServerBulkLoad#testAtomicBulkLoad: 16 threads (was 4), 327 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
regionserver.TestMasterAddressManager#testMasterAddressManagerFromZK: 10 threads (was 8), 85 file descriptors (was 79).  -thread leak?-  -file handle leak?- 
regionserver.TestMultiColumnScanner#testMultiColumnScanner[0]: 6 threads (was 4), 75 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
regionserver.TestRegionServerMetrics#testMultipleRegions: 18 threads (was 4), 118 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
regionserver.TestRpcMetrics#testCustomMetrics: 21 threads (was 9), 116 file descriptors (was 79).  -thread leak?-  -file handle leak?- 
regionserver.TestSeekOptimizations#testMultipleTimestampRanges[0]: 6 threads (was 4), 75 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
regionserver.TestServerCustomProtocol#testSingleMethod: 144 threads (was 141), 258 file descriptors (was 258).  -thread leak?- 
regionserver.TestServerCustomProtocol#testRowRange: 147 threads (was 144), 258 file descriptors (was 258).  -thread leak?- 
regionserver.TestServerCustomProtocol#testCompountCall: 150 threads (was 147), 258 file descriptors (was 258).  -thread leak?- 
regionserver.TestServerCustomProtocol#testNullCall: 153 threads (was 150), 258 file descriptors (was 258).  -thread leak?- 
regionserver.TestServerCustomProtocol#testNullReturn: 156 threads (was 153), 258 file descriptors (was 258).  -thread leak?- 
regionserver.TestServerCustomProtocol#testVoidReturnType: 159 threads (was 156), 258 file descriptors (was 258).  -thread leak?- 
regionserver.TestSplitLogWorker#testAcquireTaskAtStartup: 7 threads (was 4), 80 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitLogWorker#testRaceForTask: 9 threads (was 7), 85 file descriptors (was 80).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitLogWorker#testPreemptTask: 12 threads (was 9), 90 file descriptors (was 85).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitLogWorker#testMultipleTasks: 15 threads (was 12), 95 file descriptors (was 90).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitLogWorker#testRescan: 19 threads (was 15), 100 file descriptors (was 95).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitTransactionOnCluster#testExistingZnodeBlocksSplitAndWeRollback: 218 threads (was 162), 377 file descriptors (was 309).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitTransactionOnCluster#testShutdownFixupWhenDaughterHasSplit: 169 threads (was 168), 315 file descriptors (was 314).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitTransaction#testFailAfterPONR: 22 threads (was 21), 101 file descriptors (was 99).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitTransaction#testPrepare: 23 threads (was 22), 103 file descriptors (was 101).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitTransaction#testPrepareWithBadSplitRow: 24 threads (was 23), 105 file descriptors (was 103).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitTransaction#testPrepareWithClosedRegion: 25 threads (was 24), 107 file descriptors (was 105).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitTransaction#testWholesomeSplit: 26 threads (was 25), 113 file descriptors (was 107).  -thread leak?-  -file handle leak?- 
regionserver.TestSplitTransaction#testRollback: 27 threads (was 26), 119 file descriptors (was 113).  -thread leak?-  -file handle leak?- 
regionserver.TestStoreFileBlockCacheSummary#testBlockCacheSummary: 140 threads (was 130), 261 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogBench#testLogPerformance: 7 threads (was 4), 76 file descriptors (was 73).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLog#testSplit: 70 threads (was 70), 176 file descriptors (was 170).  -file handle leak?- 
regionserver.wal.TestHLog#Broken_testSync: 71 threads (was 70), 179 file descriptors (was 176).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLog#testAppendClose: 101 threads (was 71), 224 file descriptors (was 179).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLog#testEditAdd: 102 threads (was 101), 236 file descriptors (was 224).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLog#testAppend: 103 threads (was 102), 239 file descriptors (was 236).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLog#testVisitors: 102 threads (was 102), 239 file descriptors (was 236).  -file handle leak?- 
regionserver.wal.TestHLogSplit#testSplitFailsIfNewHLogGetsCreatedAfterSplitStarted: 56 threads (was 55), 149 file descriptors (was 143).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testSplitPreservesEdits: 58 threads (was 56), 151 file descriptors (was 149).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testEmptyLogFiles: 59 threads (was 57), 150 file descriptors (was 151).  -thread leak?- 
regionserver.wal.TestHLogSplit#testEmptyOpenLogFiles: 62 threads (was 58), 152 file descriptors (was 150).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testOpenZeroLengthReportedFileButWithDataGetsSplit: 67 threads (was 61), 166 file descriptors (was 152).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testFirstLineCorruptionLogFileSkipErrorsPasses: 73 threads (was 64), 172 file descriptors (was 153).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testMiddleGarbageCorruptionSkipErrorsReadsHalfOfFile: 73 threads (was 72), 182 file descriptors (was 172).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testTrailingGarbageCorruptionLogFileSkipErrorsFalseThrows: 72 threads (was 72), 177 file descriptors (was 176).  -file handle leak?- 
regionserver.wal.TestHLogSplit#testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs: 73 threads (was 72), 178 file descriptors (was 177).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testEOFisIgnored: 75 threads (was 73), 179 file descriptors (was 178).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testLogsGetArchivedAfterSplit: 84 threads (was 75), 192 file descriptors (was 179).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testSplitWillNotTouchLogsIfNewHLogGetsCreatedAfterSplitStarted: 78 threads (was 77), 181 file descriptors (was 178).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testSplitWillFailIfWritingToRegionFails: 83 threads (was 78), 192 file descriptors (was 181).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testSplitDeletedRegion: 86 threads (was 83), 190 file descriptors (was 192).  -thread leak?- 
regionserver.wal.TestHLogSplit#testIOEOnOutputThread: 87 threads (was 86), 188 file descriptors (was 190).  -thread leak?- 
regionserver.wal.TestHLogSplit#testMovedHLogDuringRecovery: 89 threads (was 86), 190 file descriptors (was 188).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testThreading: 94 threads (was 89), 200 file descriptors (was 190).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testThreadingSlowWriterSmallBuffer: 97 threads (was 94), 205 file descriptors (was 200).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestHLogSplit#testSplitLogFileWithOneRegion: 98 threads (was 97), 212 file descriptors (was 205).  -file handle leak?- 
regionserver.wal.TestHLogSplit#testSplitLogFileFirstLineCorruptionLog: 96 threads (was 95), 208 file descriptors (was 208).  -thread leak?- 
regionserver.wal.TestLogRollAbort#testRSAbortWithUnflushedEdits: 23 threads (was 4), 149 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestLogRolling#testLogRolling: 18 threads (was 4), 156 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestLogRolling#testLogRollOnDatanodeDeath: 23 threads (was 18), 149 file descriptors (was 156).  -thread leak?- 
regionserver.wal.TestLogRolling#testLogRollOnPipelineRestart: 43 threads (was 23), 229 file descriptors (was 149).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestWALReplay#test2727: 74 threads (was 69), 176 file descriptors (was 161).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestWALReplay#testRegionMadeOfBulkLoadedFilesOnly: 84 threads (was 74), 189 file descriptors (was 176).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestWALReplay#testReplayEditsWrittenViaHRegion: 92 threads (was 84), 208 file descriptors (was 189).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestWALReplay#testReplayEditsAfterPartialFlush: 96 threads (was 92), 213 file descriptors (was 208).  -thread leak?-  -file handle leak?- 
regionserver.wal.TestWALReplay#testReplayEditsWrittenIntoWAL: 100 threads (was 96), 214 file descriptors (was 213).  -thread leak?-  -file handle leak?- 
replication.regionserver.TestReplicationSink#testBatchSink: 267 threads (was 266), 452 file descriptors (was 452).  -thread leak?- 
replication.regionserver.TestReplicationSink#testMixedPutTables: 268 threads (was 267), 452 file descriptors (was 452).  -thread leak?- 
replication.regionserver.TestReplicationSourceManager#testLogRoll: 14 threads (was 12), 93 file descriptors (was 91).  -thread leak?-  -file handle leak?- 
replication.TestMasterReplication#testCyclicReplication: 114 threads (was 4), 334 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
replication.TestMasterReplication#testSimplePutDelete: 113 threads (was 114), 390 file descriptors (was 334).  -file handle leak?- 
replication.TestMultiSlaveReplication#testMultiSlaveReplication: 102 threads (was 14), 316 file descriptors (was 95).  -thread leak?-  -file handle leak?- 
replication.TestReplication#testSimplePutDelete: 424 threads (was 421), 683 file descriptors (was 679).  -thread leak?-  -file handle leak?- 
replication.TestReplication#testSmallBatch: 428 threads (was 424), 691 file descriptors (was 683).  -thread leak?-  -file handle leak?- 
replication.TestReplication#testAddAndRemoveClusters: 402 threads (was 398), 628 file descriptors (was 622).  -thread leak?-  -file handle leak?- 
replication.TestReplication#loadTesting: 412 threads (was 402), 655 file descriptors (was 628).  -thread leak?-  -file handle leak?- 
replication.TestReplication#testVerifyRepJob: 418 threads (was 412), 671 file descriptors (was 655).  -thread leak?-  -file handle leak?- 
replication.TestReplicationSource#testLogMoving: 42 threads (was 41), 129 file descriptors (was 125).  -thread leak?-  -file handle leak?- 
rest.client.TestRemoteAdmin#testCreateAnDeleteTable: 142 threads (was 133), 264 file descriptors (was 251).  -thread leak?-  -file handle leak?- 
rest.client.TestRemoteTable#testGetTableDescriptor: 139 threads (was 138), 260 file descriptors (was 256).  -thread leak?-  -file handle leak?- 
rest.client.TestRemoteTable#testPut: 140 threads (was 139), 260 file descriptors (was 260).  -thread leak?- 
rest.TestGzipFilter#testGzipFilter: 139 threads (was 137), 260 file descriptors (was 256).  -thread leak?-  -file handle leak?- 
rest.TestMultiRowResource#testMultiCellGetJSON: 139 threads (was 137), 260 file descriptors (was 257).  -thread leak?-  -file handle leak?- 
rest.TestRowResource#testDelete: 265 threads (was 255), 448 file descriptors (was 425).  -thread leak?-  -file handle leak?- 
rest.TestScannerResource#testSimpleScannerXML: 139 threads (was 136), 251 file descriptors (was 244).  -thread leak?-  -file handle leak?- 
rest.TestScannersWithFilters#testNoFilter: 265 threads (was 264), 442 file descriptors (was 439).  -thread leak?-  -file handle leak?- 
rest.TestSchemaResource#testTableCreateAndDeleteXML: 141 threads (was 133), 264 file descriptors (was 252).  -thread leak?-  -file handle leak?- 
rest.TestSchemaResource#testTableCreateAndDeletePB: 146 threads (was 141), 264 file descriptors (was 264).  -thread leak?- 
rest.TestStatusResource#testGetClusterStatusXML: 135 threads (was 133), 257 file descriptors (was 252).  -thread leak?-  -file handle leak?- 
rest.TestTransform#testTransform: 139 threads (was 137), 257 file descriptors (was 253).  -thread leak?-  -file handle leak?- 
rest.TestVersionResource#testGetStargateVersionText: 134 threads (was 133), 255 file descriptors (was 252).  -thread leak?-  -file handle leak?- 
rest.TestVersionResource#testGetStorageClusterVersionText: 135 threads (was 134), 257 file descriptors (was 255).  -thread leak?-  -file handle leak?- 
security.TestUser#testRunAs: 27 threads (was 27), 136 file descriptors (was 135).  -file handle leak?- 
TestAcidGuarantees#testGetAtomicity: 21 threads (was 4), 126 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
TestAcidGuarantees#testScanAtomicity: 29 threads (was 21), 180 file descriptors (was 126).  -thread leak?-  -file handle leak?- 
TestAcidGuarantees#testMixedAtomicity: 37 threads (was 29), 178 file descriptors (was 180).  -thread leak?- 
TestGlobalMemStoreSize#testGlobalMemStore: 35 threads (was 4), 191 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
TestHBaseTestingUtility#testMultiClusters: 45 threads (was 4), 190 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
TestHBaseTestingUtility#testMultipleStartStop: 47 threads (was 37), 213 file descriptors (was 178).  -thread leak?-  -file handle leak?- 
TestHBaseTestingUtility#testMiniZooKeeper: 53 threads (was 47), 229 file descriptors (was 213).  -thread leak?-  -file handle leak?- 
TestHBaseTestingUtility#testMiniDFSCluster: 51 threads (was 53), 245 file descriptors (was 229).  -file handle leak?- 
TestHBaseTestingUtility#testSetupClusterTestBuildDir: 57 threads (was 51), 261 file descriptors (was 245).  -thread leak?-  -file handle leak?- 
TestInfoServers#testInfoServersRedirect: 138 threads (was 137), 266 file descriptors (was 260).  -thread leak?-  -file handle leak?- 
TestInfoServers#testInfoServersStatusPages: 139 threads (was 138), 268 file descriptors (was 266).  -thread leak?-  -file handle leak?- 
TestMultiVersions#testTimestamps: 137 threads (was 130), 266 file descriptors (was 253).  -thread leak?-  -file handle leak?- 
TestMultiVersions#testGetRowVersions: 150 threads (was 137), 290 file descriptors (was 266).  -thread leak?-  -file handle leak?- 
TestMultiVersions#testScanMultipleVersions: 152 threads (was 151), 292 file descriptors (was 291).  -thread leak?-  -file handle leak?- 
TestRegionRebalancing#testRebalanceOnRegionServerNumberChange: 567 threads (was 130), 747 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
TestZooKeeper#testRegionServerSessionExpired: 199 threads (was 182), 347 file descriptors (was 314).  -thread leak?-  -file handle leak?- 
TestZooKeeper#testMasterSessionExpired: 204 threads (was 199), 355 file descriptors (was 347).  -thread leak?-  -file handle leak?- 
TestZooKeeper#testMultipleZK: 207 threads (was 204), 362 file descriptors (was 355).  -thread leak?-  -file handle leak?- 
TestZooKeeper#testZNodeDeletes: 209 threads (was 207), 367 file descriptors (was 362).  -thread leak?-  -file handle leak?- 
TestZooKeeper#testCreateSilentIsReallySilent: 212 threads (was 209), 376 file descriptors (was 367).  -thread leak?-  -file handle leak?- 
thrift2.TestThriftHBaseServiceHandler#testExists: 135 threads (was 134), 252 file descriptors (was 252).  -thread leak?- 
thrift2.TestThriftHBaseServiceHandler#testPutGet: 136 threads (was 135), 252 file descriptors (was 252).  -thread leak?- 
thrift2.TestThriftHBaseServiceHandler#testPutGetMultiple: 137 threads (was 136), 252 file descriptors (was 252).  -thread leak?- 
thrift2.TestThriftHBaseServiceHandler#testDeleteMultiple: 138 threads (was 137), 252 file descriptors (was 252).  -thread leak?- 
thrift2.TestThriftHBaseServiceHandler#testDelete: 139 threads (was 138), 252 file descriptors (was 252).  -thread leak?- 
thrift2.TestThriftHBaseServiceHandler#testDeleteAllTimestamps: 140 threads (was 139), 252 file descriptors (was 252).  -thread leak?- 
thrift2.TestThriftHBaseServiceHandler#testDeleteSingleTimestamp: 141 threads (was 140), 252 file descriptors (was 252).  -thread leak?- 
thrift2.TestThriftHBaseServiceHandler#testIncrement: 142 threads (was 141), 252 file descriptors (was 252).  -thread leak?- 
thrift2.TestThriftHBaseServiceHandler#testCheckAndPut: 143 threads (was 142), 252 file descriptors (was 252).  -thread leak?- 
thrift2.TestThriftHBaseServiceHandler#testCheckAndDelete: 144 threads (was 143), 252 file descriptors (was 252).  -thread leak?- 
thrift2.TestThriftHBaseServiceHandler#testScan: 145 threads (was 144), 252 file descriptors (was 252).  -thread leak?- 
thrift.TestThriftServerCmdLine#testRunThriftServer[0]: 136 threads (was 130), 260 file descriptors (was 253).  -thread leak?-  -file handle leak?- 
thrift.TestThriftServerCmdLine#testRunThriftServer[1]: 136 threads (was 136), 264 file descriptors (was 260).  -file handle leak?- 
thrift.TestThriftServerCmdLine#testRunThriftServer[2]: 136 threads (was 136), 269 file descriptors (was 264).  -file handle leak?- 
thrift.TestThriftServerCmdLine#testRunThriftServer[3]: 136 threads (was 136), 274 file descriptors (was 269).  -file handle leak?- 
thrift.TestThriftServerCmdLine#testRunThriftServer[4]: 136 threads (was 136), 278 file descriptors (was 274).  -file handle leak?- 
thrift.TestThriftServerCmdLine#testRunThriftServer[6]: 132 threads (was 132), 277 file descriptors (was 272).  -file handle leak?- 
thrift.TestThriftServerCmdLine#testRunThriftServer[7]: 132 threads (was 132), 282 file descriptors (was 277).  -file handle leak?- 
thrift.TestThriftServerCmdLine#testRunThriftServer[9]: 133 threads (was 132), 284 file descriptors (was 282).  -thread leak?-  -file handle leak?- 
thrift.TestThriftServerCmdLine#testRunThriftServer[13]: 133 threads (was 132), 284 file descriptors (was 282).  -thread leak?-  -file handle leak?- 
thrift.TestThriftServer#doTestTableCreateDrop: 140 threads (was 130), 255 file descriptors (was 253).  -thread leak?-  -file handle leak?- 
util.hbck.TestOfflineMetaRebuildBase#testMetaRebuild: 36 threads (was 4), 215 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
util.hbck.TestOfflineMetaRebuildHole#testMetaRebuildHoleFail: 33 threads (was 4), 212 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
util.hbck.TestOfflineMetaRebuildOverlap#testMetaRebuildOverlapFail: 35 threads (was 4), 203 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
util.TestFSTableDescriptors#testCreateAndUpdate: 4 threads (was 4), 74 file descriptors (was 70).  -file handle leak?- 
util.TestFSUtils#testIsHDFS: 8 threads (was 4), 101 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
util.TestFSUtils#testcomputeHDFSBlocksDistribution: 28 threads (was 8), 208 file descriptors (was 101).  -thread leak?-  -file handle leak?- 
util.TestIdLock#testMultipleClients: 129 threads (was 4), 70 file descriptors (was 70).  -thread leak?- 
util.TestMergeTable#testMergeTable: 18 threads (was 4), 152 file descriptors (was 70).  -thread leak?-  -file handle leak?- 
util.TestRegionSplitter#testCreatePresplitTableHex: 138 threads (was 130), 255 file descriptors (was 250).  -thread leak?-  -file handle leak?- 
zookeeper.TestZKTable#testTableStates: 10 threads (was 8), 85 file descriptors (was 79).  -thread leak?-  -file handle leak?- 
zookeeper.TestZooKeeperNodeTracker#testInterruptible: 10 threads (was 8), 85 file descriptors (was 79).  -thread leak?-  -file handle leak?- 
zookeeper.TestZooKeeperNodeTracker#testNodeTracker: 14 threads (was 10), 95 file descriptors (was 85).  -thread leak?-  -file handle leak?- 

                
> Monitor the open file descriptors and the threads counters during the unit tests
> --------------------------------------------------------------------------------
>
>                 Key: HBASE-4965
>                 URL: https://issues.apache.org/jira/browse/HBASE-4965
>             Project: HBase
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.94.0
>         Environment: all
>            Reporter: nkeywal
>            Assignee: nkeywal
>            Priority: Minor
>         Attachments: 4965_all.patch, ResourceChecker.java, ResourceCheckerJUnitRule.java
>
>
> We're seeing a lot of issues with hadoop-qa related to threads or file descriptors.
> Monitoring these counters would ease the analysis.
> Note as well that
>  - if we want to execute the tests in the same jvm (because the test is small or because we want to share the cluster) we can't afford to leak too many resources
>  - if the tests leak, it's more difficult to detect a leak in the software itself.
> I attach piece of code that I used. It requires two lines in a unit test class to:
> - before every test, count the threads and the open file descriptor
> - after every test, compare with the previous value.
> I ran it on some tests; we have for example:
> - client.TestMultiParallel#testBatchWithManyColsInOneRowGetAndPut: 232 threads (was 231), 390 file descriptors (was 390). => TestMultiParallel uses 232 threads!
> - client.TestMultipleTimestamps#testWithColumnDeletes: 152 threads (was 151), 283 file descriptors (was 282).
> - client.TestAdmin#testCheckHBaseAvailableClosesConnection: 477 threads (was 294), 815 file descriptors (was 461)
> - client.TestMetaMigrationRemovingHTD#testMetaMigration: 149 threads (was 148), 310 file descriptors (was 307).
> It's not always leaks, we can expect some pooling effects. But still...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message