hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Johansville <james.johansvi...@gmail.com>
Subject Inconsistent scan performance
Date Thu, 24 Mar 2016 23:45:13 GMT
Hello all,

So, I wrote a Java application for HBase that does a partitioned full-table
scan according to a set number of partitions. For example, if there are 20
partitions specified, then 20 separate full scans are launched that cover
an equal slice of the row identifier range.

The rows are uniformly distributed throughout the RegionServers. I
confirmed this through the hbase shell. I have only one column family, and
each row has the same number of column qualifiers.

My problem is that the individual scan performance is wildly inconsistent
even though they fetch approximately a similar number of rows. This
inconsistency appears to be random with respect to hosts or regionservers
or partitions or CPU cores. I am the only user of the fleet and not running
any other concurrent HBase operation.

I started measuring from the beginning of the scan and stopped measuring
after the scan was completed. I am not doing any logic with the results,
just scanning them.

For ~230K rows fetched per scan, I am getting anywhere from 4 seconds to
100+ seconds. This seems a little too bouncy for me. Does anyone have any
insight? By comparison, a similar utility I wrote to upsert to
regionservers was very consistent in ops/sec and I had no issues with it.

Using 13 partitions on a machine that has 32 CPU cores and 16 GB heap, I
see anywhere between 3K ops/sec to 82K ops/sec. Here's an example of log
output I saved that used 130 partitions.

total # partitions:130; partition id:47; rows:232730 elapsed_sec:6.401
total # partitions:130; partition id:100; rows:206890 elapsed_sec:6.636
total # partitions:130; partition id:63; rows:233437 elapsed_sec:7.586
total # partitions:130; partition id:9; rows:232585 elapsed_sec:32.985
total # partitions:130; partition id:19; rows:234192 elapsed_sec:38.733
total # partitions:130; partition id:1; rows:232860 elapsed_sec:48.479
total # partitions:130; partition id:125; rows:205334 elapsed_sec:41.911
total # partitions:130; partition id:123; rows:206622 elapsed_sec:42.281
total # partitions:130; partition id:54; rows:232811 elapsed_sec:49.083

I use setCacheBlocks(false), setCaching(5000).  Does anyone have any
insight into how I can make the read performance more consistent?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message