hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-13291) Lift the scan ceiling
Date Thu, 26 Mar 2015 19:20:53 GMT

     [ https://issues.apache.org/jira/browse/HBASE-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

stack updated HBASE-13291:
    Attachment: traces.small2.svg
                Screen Shot 2015-03-26 at 12.12.13 PM.png

Minor update. 

Played with doubling Handlers and Readers but made no difference in throughtput. I was maybe
burning more CPU.  Using 20 handlers instead of 30 (default) had us doing about same cpu but
less throughput (7k vs 8k). So we are up against ceiling.

Looking at what we were doing, most is compareTo. I tried using java compareTo instead of
our copy of guava unsafe (our default) but pure java was more cpu and slower by about 15%
(8k vs 7k).

Attached flamegraph is where we are at when up against the ceiling. Has inlining which gets
us some small benefit but means lower methods are missing. Got perf top working using https://github.com/jrudolph/perf-map-agent.
It just shows us same thing that we spend all our CPU in HFileReaderV3#ScannerV3#readKeyValueLen
and SQM#match (compareTo). If I ask perf top to show inlined functions, it fingers the HBB#_get
again. Hmm...

> Lift the scan ceiling
> ---------------------
>                 Key: HBASE-13291
>                 URL: https://issues.apache.org/jira/browse/HBASE-13291
>             Project: HBase
>          Issue Type: Improvement
>          Components: Scanners
>    Affects Versions: 1.0.0
>            Reporter: stack
>            Assignee: stack
>         Attachments: 13291.inlining.txt, Screen Shot 2015-03-26 at 12.12.13 PM.png, q
(1).png, traces.7.svg, traces.filterall.svg, traces.nofilter.svg, traces.small2.svg, traces.smaller.svg
> Scanning medium sized rows with multiple concurrent scanners exhibits interesting 'ceiling'
properties. A server runs at about 6.7k ops a second using 450% of possible 1600% of CPUs
 when 4 clients each with 10 threads doing scan 1000 rows.  If I add '--filterAll' argument
(do not return results), then we run at 1450% of possible 1600% possible but we do 8k ops
a second.
> Let me attach flame graphs for two cases. Unfortunately, there is some frustrating dark
art going on. Let me try figure it... Filing issue in meantime to keep score in.

This message was sent by Atlassian JIRA

View raw message