hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elliott Clark (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
Date Fri, 28 Jun 2013 20:51:22 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13695774#comment-13695774 ] 

Elliott Clark edited comment on HBASE-6295 at 6/28/13 8:50 PM:
---------------------------------------------------------------

So the good news is this patch really helped our perf on YCSB.
Build #91 is where this patch went in.

The bad news is that the addendum didn't fix integration tests on a real cluster.  They still fail.

Here's some logs for integration-test-data-ingest-slow-deterministic

{code}
2013-06-28 13:08:32,882 DEBUG [hbase-table-pool-175-thread-2] client.AsyncProcess: Attempt #10/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: b7040e878147caf1f6de338faad504be
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 3205 ms.
2013-06-28 13:08:32,939 DEBUG [hbase-table-pool-173-thread-1] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:32,945 DEBUG [hbase-table-pool-173-thread-2] client.AsyncProcess: Attempt #10/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,22222220,1372449851892.c8a2a77e0690901df245ac9fff088a0e., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c8a2a77e0690901df245ac9fff088a0e
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 3201 ms.
2013-06-28 13:08:32,996 DEBUG [hbase-table-pool-174-thread-1] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,00000000000000
2013-06-28 13:08:33,030 DEBUG [hbase-table-pool-174-thread-1] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:33,032 DEBUG [hbase-table-pool-174-thread-2] client.AsyncProcess: Attempt #10/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: b7040e878147caf1f6de338faad504be
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 3201 ms.
2013-06-28 13:08:33,371 DEBUG [hbase-table-pool-177-thread-2] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:33,408 DEBUG [hbase-table-pool-177-thread-2] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:33,411 DEBUG [hbase-table-pool-171-thread-1] client.AsyncProcess: Attempt #10/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c6df99964212adf9c8d9807926371a57
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 3215 ms.
2013-06-28 13:08:33,411 DEBUG [hbase-table-pool-177-thread-1] client.AsyncProcess: Attempt #10/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c6df99964212adf9c8d9807926371a57
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 3212 ms.
2013-06-28 13:08:35,941 DEBUG [hbase-table-pool-180-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: b7040e878147caf1f6de338faad504be
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6460 ms.
2013-06-28 13:08:35,956 DEBUG [hbase-table-pool-178-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,3bbbbbb8,1372449851893.ebd0ab2aba6df7f8ed2ee5e47872dd36., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=53019, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: ebd0ab2aba6df7f8ed2ee5e47872dd36
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6410 ms.
2013-06-28 13:08:35,968 DEBUG [hbase-table-pool-172-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,66666660,1372449851893.6910db29befeea246245b02808ac45ec., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: 6910db29befeea246245b02808ac45ec
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6421 ms.
2013-06-28 13:08:36,000 DEBUG [hbase-table-pool-176-thread-2] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:36,051 DEBUG [hbase-table-pool-176-thread-2] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:36,053 DEBUG [hbase-table-pool-176-thread-1] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c6df99964212adf9c8d9807926371a57
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6456 ms.
2013-06-28 13:08:36,054 DEBUG [hbase-table-pool-179-thread-1] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:36,061 DEBUG [hbase-table-pool-179-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,22222220,1372449851892.c8a2a77e0690901df245ac9fff088a0e., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c8a2a77e0690901df245ac9fff088a0e
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6422 ms.
2013-06-28 13:08:36,088 DEBUG [hbase-table-pool-175-thread-1] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: b7040e878147caf1f6de338faad504be
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6445 ms.
2013-06-28 13:08:36,150 DEBUG [hbase-table-pool-173-thread-2] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:36,156 DEBUG [hbase-table-pool-173-thread-1] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,22222220,1372449851892.c8a2a77e0690901df245ac9fff088a0e., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c8a2a77e0690901df245ac9fff088a0e
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6459 ms.
2013-06-28 13:08:36,237 DEBUG [hbase-table-pool-174-thread-2] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,00000000000000
2013-06-28 13:08:36,273 DEBUG [hbase-table-pool-174-thread-2] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:36,276 DEBUG [hbase-table-pool-174-thread-1] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: b7040e878147caf1f6de338faad504be
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6441 ms.
2013-06-28 13:08:36,627 DEBUG [hbase-table-pool-177-thread-1] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:36,663 DEBUG [hbase-table-pool-177-thread-1] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:36,666 DEBUG [hbase-table-pool-171-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c6df99964212adf9c8d9807926371a57
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6415 ms.
2013-06-28 13:08:36,667 DEBUG [hbase-table-pool-177-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c6df99964212adf9c8d9807926371a57
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6453 ms.
2013-06-28 13:08:37,368 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=6575, cols=68.0 K, time=00:00:50 Overall: [keys/s= 131, latency=46 ms], insertedUpTo=74999
2013-06-28 13:08:42,367 INFO  [HBaseWriterThread_7] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=6934, tasksDone=6933, currentTasksDone=6933, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,368 DEBUG [hbase-table-pool-178-thread-1] client.AsyncProcess: Attempt #12/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,3bbbbbb8,1372449851893.ebd0ab2aba6df7f8ed2ee5e47872dd36., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=53019, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: ebd0ab2aba6df7f8ed2ee5e47872dd36
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6401 ms.
2013-06-28 13:08:42,369 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=6575, cols=68.0 K, time=00:00:55 Overall: [keys/s= 119, latency=46 ms], insertedUpTo=74999
2013-06-28 13:08:42,389 INFO  [HBaseWriterThread_1] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=6883, tasksDone=6882, currentTasksDone=6882, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,390 DEBUG [hbase-table-pool-172-thread-1] client.AsyncProcess: Attempt #12/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,66666660,1372449851893.6910db29befeea246245b02808ac45ec., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: 6910db29befeea246245b02808ac45ec
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6432 ms.
2013-06-28 13:08:42,401 INFO  [HBaseWriterThread_9] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=6886, tasksDone=6885, currentTasksDone=6885, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,486 DEBUG [hbase-table-pool-179-thread-2] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:42,491 INFO  [HBaseWriterThread_8] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=7006, tasksDone=7005, currentTasksDone=7005, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,497 DEBUG [HBaseWriterThread_9] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:42,533 DEBUG [HBaseWriterThread_9] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:42,534 INFO  [HBaseWriterThread_4] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=7105, tasksDone=7104, currentTasksDone=7104, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,534 INFO  [HBaseWriterThread_5] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=6812, tasksDone=6811, currentTasksDone=6811, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,615 INFO  [HBaseWriterThread_2] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=7169, tasksDone=7168, currentTasksDone=7168, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,739 INFO  [HBaseWriterThread_3] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=7091, tasksDone=7090, currentTasksDone=7090, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:43,081 INFO  [HBaseWriterThread_0] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=6907, tasksDone=6906, currentTasksDone=6906, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:43,120 INFO  [HBaseWriterThread_6] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=7102, tasksDone=7101, currentTasksDone=7101, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:47,370 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=8191, cols=84.6 K, time=00:01:00 Overall: [keys/s= 136, latency=55 ms] Current: [keys/s=323, latency=91 ms], insertedUpTo=74999
2013-06-28 13:08:52,370 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=10203, cols=105.5 K, time=00:01:05 Overall: [keys/s= 156, latency=53 ms] Current: [keys/s=402, latency=43 ms], insertedUpTo=74999
2013-06-28 13:08:57,371 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=12404, cols=127.8 K, time=00:01:10 Overall: [keys/s= 177, latency=47 ms] Current: [keys/s=440, latency=22 ms], insertedUpTo=74999
2013-06-28 13:09:02,373 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=14290, cols=146.8 K, time=00:01:15 Overall: [keys/s= 190, latency=44 ms] Current: [keys/s=377, latency=26 ms], insertedUpTo=74999
Failed to write keys: 4
Failed to write key: 81469
Failed to write key: 81503
Failed to write key: 81518
Failed to write key: 81524
2013-06-28 13:09:04,376 ERROR [main] hbase.IngestIntegrationTestBase: Load failed with error code 1
2013-06-28 13:10:31,691 INFO  [Thread-4] util.ChaosMonkey: Sleeping for 84975 to add jitter
2013-06-28 13:11:56,667 DEBUG [main] hbase.IngestIntegrationTestBase: Restoring the cluster
2013-06-28 13:11:56,679 DEBUG [main] hbase.IngestIntegrationTestBase: Done restoring the cluster
E
Time: 466.629
There was 1 failure:
1) testDataIngest(org.apache.hadoop.hbase.IntegrationTestDataIngestSlowDeterministic)
junit.framework.AssertionFailedError: Load failed with error code 1
	at junit.framework.Assert.fail(Assert.java:57)
	at org.apache.hadoop.hbase.IngestIntegrationTestBase.runIngestTest(IngestIntegrationTestBase.java:114)
	at org.apache.hadoop.hbase.IntegrationTestDataIngestSlowDeterministic.testDataIngest(IntegrationTestDataIngestSlowDeterministic.java:72)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runners.Suite.runChild(Suite.java:127)
	at org.junit.runners.Suite.runChild(Suite.java:26)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:117)
	at org.apache.hadoop.hbase.IntegrationTestsDriver.doWork(IntegrationTestsDriver.java:111)
	at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:108)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{code}
                
      was (Author: eclark):
    So the good news is this patch really helped our perf on YCSB.
Build #91 is where this patch went in.

The bad news is that the addendum didn't fix integration tests on a real cluster.  They still fail.

{code}
2013-06-28 13:08:32,882 DEBUG [hbase-table-pool-175-thread-2] client.AsyncProcess: Attempt #10/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: b7040e878147caf1f6de338faad504be
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 3205 ms.
2013-06-28 13:08:32,939 DEBUG [hbase-table-pool-173-thread-1] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:32,945 DEBUG [hbase-table-pool-173-thread-2] client.AsyncProcess: Attempt #10/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,22222220,1372449851892.c8a2a77e0690901df245ac9fff088a0e., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c8a2a77e0690901df245ac9fff088a0e
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 3201 ms.
2013-06-28 13:08:32,996 DEBUG [hbase-table-pool-174-thread-1] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,00000000000000
2013-06-28 13:08:33,030 DEBUG [hbase-table-pool-174-thread-1] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:33,032 DEBUG [hbase-table-pool-174-thread-2] client.AsyncProcess: Attempt #10/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: b7040e878147caf1f6de338faad504be
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 3201 ms.
2013-06-28 13:08:33,371 DEBUG [hbase-table-pool-177-thread-2] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:33,408 DEBUG [hbase-table-pool-177-thread-2] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:33,411 DEBUG [hbase-table-pool-171-thread-1] client.AsyncProcess: Attempt #10/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c6df99964212adf9c8d9807926371a57
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 3215 ms.
2013-06-28 13:08:33,411 DEBUG [hbase-table-pool-177-thread-1] client.AsyncProcess: Attempt #10/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c6df99964212adf9c8d9807926371a57
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 3212 ms.
2013-06-28 13:08:35,941 DEBUG [hbase-table-pool-180-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: b7040e878147caf1f6de338faad504be
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6460 ms.
2013-06-28 13:08:35,956 DEBUG [hbase-table-pool-178-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,3bbbbbb8,1372449851893.ebd0ab2aba6df7f8ed2ee5e47872dd36., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=53019, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: ebd0ab2aba6df7f8ed2ee5e47872dd36
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6410 ms.
2013-06-28 13:08:35,968 DEBUG [hbase-table-pool-172-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,66666660,1372449851893.6910db29befeea246245b02808ac45ec., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: 6910db29befeea246245b02808ac45ec
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6421 ms.
2013-06-28 13:08:36,000 DEBUG [hbase-table-pool-176-thread-2] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:36,051 DEBUG [hbase-table-pool-176-thread-2] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:36,053 DEBUG [hbase-table-pool-176-thread-1] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c6df99964212adf9c8d9807926371a57
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6456 ms.
2013-06-28 13:08:36,054 DEBUG [hbase-table-pool-179-thread-1] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:36,061 DEBUG [hbase-table-pool-179-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,22222220,1372449851892.c8a2a77e0690901df245ac9fff088a0e., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c8a2a77e0690901df245ac9fff088a0e
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6422 ms.
2013-06-28 13:08:36,088 DEBUG [hbase-table-pool-175-thread-1] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: b7040e878147caf1f6de338faad504be
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6445 ms.
2013-06-28 13:08:36,150 DEBUG [hbase-table-pool-173-thread-2] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:36,156 DEBUG [hbase-table-pool-173-thread-1] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,22222220,1372449851892.c8a2a77e0690901df245ac9fff088a0e., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c8a2a77e0690901df245ac9fff088a0e
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6459 ms.
2013-06-28 13:08:36,237 DEBUG [hbase-table-pool-174-thread-2] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,00000000000000
2013-06-28 13:08:36,273 DEBUG [hbase-table-pool-174-thread-2] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:36,276 DEBUG [hbase-table-pool-174-thread-1] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,bbbbbbb0,1372449851893.b7040e878147caf1f6de338faad504be., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: b7040e878147caf1f6de338faad504be
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6441 ms.
2013-06-28 13:08:36,627 DEBUG [hbase-table-pool-177-thread-1] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:36,663 DEBUG [hbase-table-pool-177-thread-1] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:36,666 DEBUG [hbase-table-pool-171-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c6df99964212adf9c8d9807926371a57
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6415 ms.
2013-06-28 13:08:36,667 DEBUG [hbase-table-pool-177-thread-2] client.AsyncProcess: Attempt #11/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,b3333328,1372449851893.c6df99964212adf9c8d9807926371a57., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: c6df99964212adf9c8d9807926371a57
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6453 ms.
2013-06-28 13:08:37,368 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=6575, cols=68.0 K, time=00:00:50 Overall: [keys/s= 131, latency=46 ms], insertedUpTo=74999
2013-06-28 13:08:42,367 INFO  [HBaseWriterThread_7] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=6934, tasksDone=6933, currentTasksDone=6933, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,368 DEBUG [hbase-table-pool-178-thread-1] client.AsyncProcess: Attempt #12/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,3bbbbbb8,1372449851893.ebd0ab2aba6df7f8ed2ee5e47872dd36., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=53019, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: ebd0ab2aba6df7f8ed2ee5e47872dd36
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6401 ms.
2013-06-28 13:08:42,369 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=6575, cols=68.0 K, time=00:00:55 Overall: [keys/s= 119, latency=46 ms], insertedUpTo=74999
2013-06-28 13:08:42,389 INFO  [HBaseWriterThread_1] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=6883, tasksDone=6882, currentTasksDone=6882, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,390 DEBUG [hbase-table-pool-172-thread-1] client.AsyncProcess: Attempt #12/14 failed for 1 operations on server a1806.halxg.cloudera.com,60020,1372449350703, resubmitting 1, tableName=IntegrationTestDataIngestSlowDeterministic, location=region=IntegrationTestDataIngestSlowDeterministic,66666660,1372449851893.6910db29befeea246245b02808ac45ec., hostname=a1806.halxg.cloudera.com,60020,1372449350703, seqNum=1, last exception was: org.apache.hadoop.hbase.exceptions.NotServingRegionException: org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: 6910db29befeea246245b02808ac45ec
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2565)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3852)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3188)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20938)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122)
	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829)
 - sleeping 6432 ms.
2013-06-28 13:08:42,401 INFO  [HBaseWriterThread_9] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=6886, tasksDone=6885, currentTasksDone=6885, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,486 DEBUG [hbase-table-pool-179-thread-2] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,22222220,00000000000000
2013-06-28 13:08:42,491 INFO  [HBaseWriterThread_8] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=7006, tasksDone=7005, currentTasksDone=7005, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,497 DEBUG [HBaseWriterThread_9] client.ClientScanner: Scan table=.META., startRow=IntegrationTestDataIngestSlowDeterministic,b3333328,00000000000000
2013-06-28 13:08:42,533 DEBUG [HBaseWriterThread_9] client.ClientScanner: Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
2013-06-28 13:08:42,534 INFO  [HBaseWriterThread_4] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=7105, tasksDone=7104, currentTasksDone=7104, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,534 INFO  [HBaseWriterThread_5] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=6812, tasksDone=6811, currentTasksDone=6811, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,615 INFO  [HBaseWriterThread_2] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=7169, tasksDone=7168, currentTasksDone=7168, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:42,739 INFO  [HBaseWriterThread_3] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=7091, tasksDone=7090, currentTasksDone=7090, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:43,081 INFO  [HBaseWriterThread_0] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=6907, tasksDone=6906, currentTasksDone=6906, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:43,120 INFO  [HBaseWriterThread_6] client.AsyncProcess: : Waiting for the global number of running tasks to be equals or less than 0, tasksSent=7102, tasksDone=7101, currentTasksDone=7101, tableName=IntegrationTestDataIngestSlowDeterministic
2013-06-28 13:08:47,370 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=8191, cols=84.6 K, time=00:01:00 Overall: [keys/s= 136, latency=55 ms] Current: [keys/s=323, latency=91 ms], insertedUpTo=74999
2013-06-28 13:08:52,370 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=10203, cols=105.5 K, time=00:01:05 Overall: [keys/s= 156, latency=53 ms] Current: [keys/s=402, latency=43 ms], insertedUpTo=74999
2013-06-28 13:08:57,371 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=12404, cols=127.8 K, time=00:01:10 Overall: [keys/s= 177, latency=47 ms] Current: [keys/s=440, latency=22 ms], insertedUpTo=74999
2013-06-28 13:09:02,373 INFO  [Thread-399] util.MultiThreadedAction: [W:10] Keys=14290, cols=146.8 K, time=00:01:15 Overall: [keys/s= 190, latency=44 ms] Current: [keys/s=377, latency=26 ms], insertedUpTo=74999
Failed to write keys: 4
Failed to write key: 81469
Failed to write key: 81503
Failed to write key: 81518
Failed to write key: 81524
2013-06-28 13:09:04,376 ERROR [main] hbase.IngestIntegrationTestBase: Load failed with error code 1
2013-06-28 13:10:31,691 INFO  [Thread-4] util.ChaosMonkey: Sleeping for 84975 to add jitter
2013-06-28 13:11:56,667 DEBUG [main] hbase.IngestIntegrationTestBase: Restoring the cluster
2013-06-28 13:11:56,679 DEBUG [main] hbase.IngestIntegrationTestBase: Done restoring the cluster
E
Time: 466.629
There was 1 failure:
1) testDataIngest(org.apache.hadoop.hbase.IntegrationTestDataIngestSlowDeterministic)
junit.framework.AssertionFailedError: Load failed with error code 1
	at junit.framework.Assert.fail(Assert.java:57)
	at org.apache.hadoop.hbase.IngestIntegrationTestBase.runIngestTest(IngestIntegrationTestBase.java:114)
	at org.apache.hadoop.hbase.IntegrationTestDataIngestSlowDeterministic.testDataIngest(IntegrationTestDataIngestSlowDeterministic.java:72)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runners.Suite.runChild(Suite.java:127)
	at org.junit.runners.Suite.runChild(Suite.java:26)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:117)
	at org.apache.hadoop.hbase.IntegrationTestsDriver.doWork(IntegrationTestsDriver.java:111)
	at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:108)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{code}
                  
> Possible performance improvement in client batch operations: presplit and send in background
> --------------------------------------------------------------------------------------------
>
>                 Key: HBASE-6295
>                 URL: https://issues.apache.org/jira/browse/HBASE-6295
>             Project: HBase
>          Issue Type: Improvement
>          Components: Client, Performance
>    Affects Versions: 0.95.2
>            Reporter: Nicolas Liochon
>            Assignee: Nicolas Liochon
>              Labels: noob
>             Fix For: 0.98.0, 0.95.2
>
>         Attachments: 6295.addendum.patch, 6295.v11.patch, 6295.v12.patch, 6295.v14.patch, 6295.v15.patch, 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 6295.v4.patch, 6295.v5.patch, 6295.v6.patch, 6295.v8.patch, 6295.v9.patch, hbase-ycsb-workloads Build time trend.png
>
>
> today batch algo is:
> {noformat}
> for Operation o: List<Op>{
>   add o to todolist
>   if todolist > maxsize or o last in list
>     split todolist per location
>     send split lists to region servers
>     clear todolist
>     wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is enough data for a single location
> It would be:
> {noformat}
> for Operation o: List<Op>{
>   get location
>   add o to todo location.todolist
>   if (location.todolist > maxLocationSize)
>     send location.todolist to region server 
>     clear location.todolist
>     // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message