hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-15619) Performance regression observed: Empty random read(get) performance of branch-1 worse than 0.98
Date Tue, 12 Apr 2016 00:27:25 GMT

     [ https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-15619:
--------------------------
    Attachment: flamegraph-108588.098.svg
                flamegraph-1221.branch-1.svg
                flamegraph-135684.1.1.svg

Flame graphs from when were were doing the C` workload where we were mostly random reading
values with about 40% of the time reading nothing from hbase.

I don't think these help  much. The shape of the server has changed too much. Would be more
useful if they'd been for the empty table. Then we might see where they are spending the extra
time... though flight recording might give better detail. I didn't do this. Let me know if
you want me to (you'd probably be better-off doing it yourself [~carp84] if bad performance
reading an empty table is important for you). 

> Performance regression observed: Empty random read(get) performance of branch-1 worse
than 0.98
> -----------------------------------------------------------------------------------------------
>
>                 Key: HBASE-15619
>                 URL: https://issues.apache.org/jira/browse/HBASE-15619
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Yu Li
>            Assignee: Yu Li
>         Attachments: compare.png, flamegraph-108588.098.svg, flamegraph-1221.branch-1.svg,
flamegraph-135684.1.1.svg
>
>
> As titled, I observed the perf regression in the final stress testing before upgrading
our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is the most important
perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>      1) 1.1.2 with important perf fixes/improvements including HBASE-15031 and HBASE-14465
>      2) 1.1.4 release
>      3) 1.2.1RC1
> 2. Test environment
>     * YCSB: 0.7.0 with [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
>     * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 100 threads
>     * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
>     * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same hardware
for client and server
> 3. Test cases
>     * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
>     * case #1: read against empty table
>     * -case #2: lrucache 100% hit-
>     * -case #3: BLOCKCACHE=>false-
> 4. Test result
> * 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so will only
paste comparison data of 0.98.12+ and 1.1.2+
> * per-RS Throughput(ops/s)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|383562|-257493-|-47594-|
> |1.1.2+|363050|-232757-|-35872-|
> * AverageLatency(us)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|2774|-4134-|-22371-|
> |1.1.2+|2930|-4572-|-29690-|
> It seems there's perf regression on RPCServer (we tried 0.98 client against 1.x server
and observed a similar perf to 1.x client)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message