hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kim Hamilton <kimdhamil...@gmail.com>
Subject endpoint coprocessor performance
Date Tue, 05 Mar 2013 01:14:26 GMT
Hi all,
I've been lurking here for a while, so thanks for all the valuable tips and
guidance you've given so far.

I'm running some experiments to understand where to use coprocessors. One
interesting scenario is computing distinct values. I ran performance tests
with two distinct value implementations: one using endpoint coprocessors,
and one using just scans (computing distinct values client side only). I
noticed that the endpoint coprocessor implementation averaged 80 ms slower
than the scan implementation. Details of that are below for anyone
interested.

To drill into the performance, I instrumented the code and ultimately
deployed a no-op endpoint coprocessor, to look at the overhead of simply
calling it. I'm measuring around 100ms for calling my empty, no-op endpoint
coprocessor.

I need to do more tests, but I believe my tests are leading me to similar
conclusions drawn here:
http://hbase-coprocessor-experiments.blogspot.com/2011/05/extending.html

I.e. if the query/scan is selective enough (I'll go out on a limb and
estimate 50-100 rows), then it's better to just perform a scan and compute
client side. Endpoint coprocessors will make sense for larger result sets
and/or scans that hit multiple regions.

Before going too far, I wanted to check if anyone in this group has
suggestions. I.e. perhaps there are just some configuration options I've
not uncovered. Does this 100ms latency sound correct?

Thanks,
Kim


*Detailed results of distinct value comparison, just FYI*

Using 0.92.1-cdh4.1.0
Scan result size ~50-100
Row size 1kb, but after filtering for only desired columns, 380 bytes

*with coprocessors*
AverageLatency(ms), 176.1353
MinLatency(ms), 42
MaxLatency(ms), 2368
95thPercentileLatency(ms), 321
99thPercentileLatency(ms), 422

*scan-only, compute distinct values client side*
AverageLatency(ms), 92.8165
MinLatency(ms), 4
MaxLatency(ms), 986
95thPercentileLatency(ms), 253
99thPercentileLatency(ms), 356

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message