hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15213) Fix increment performance regression caused by HBASE-8763 on branch-1.0
Date Sat, 06 Feb 2016 00:39:39 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135353#comment-15135353
] 

stack commented on HBASE-15213:
-------------------------------

Here is a run of 1.2:

{code}
2016-02-05 16:33:35,216 INFO  [pool-1-thread-3-SendThread(localhost:2181)] zookeeper.ClientCnxn:
Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x152b3feb54b0007,
negotiated timeout = 40000
2016-02-05 16:34:34,248 INFO  [main] hbase.IncrementPerformanceTest: 75th=5.6205, 95th=10.131799999999998,
99th=27.87007000000001
{code}

... which confirms what [~junegunn] says about 1.2.0 not needing the hbase.increment.fast.but.narrow.consistency
stuff (dang... it took me a while to forward port HBASE-15091)

> Fix increment performance regression caused by HBASE-8763 on branch-1.0
> -----------------------------------------------------------------------
>
>                 Key: HBASE-15213
>                 URL: https://issues.apache.org/jira/browse/HBASE-15213
>             Project: HBase
>          Issue Type: Bug
>          Components: Performance
>            Reporter: Junegunn Choi
>            Assignee: Junegunn Choi
>         Attachments: 15157v3.branch-1.1.patch, HBASE-15213-increment.png, HBASE-15213.branch-1.0.patch,
HBASE-15213.v1.branch-1.0.patch
>
>
> This is an attempt to fix the increment performance regression caused by HBASE-8763 on
branch-1.0.
> I'm aware that hbase.increment.fast.but.narrow.consistency was added to branch-1.0 (HBASE-15031)
to address the issue and a separate work is ongoing on master branch, but anyway, this is
my take on the problem.
> I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what caused the slowdown
but I could indeed reproduce the performance regression.
> Test setup:
> - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers, JDK 1.7)
> - Client: Another box of the same spec
> - Increments on random 10k records on a single-region table, recreated every time
> Increment throughput (TPS):
> || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0 (narrow-consistency)
||
> || 1            | 2661                         | 2486        | 2359  |
> || 2            | 5048                         | 5064        | 4867  |
> || 4            | 7503                         | 8071        | 8690  |
> || 8            | 10471                        | 10886       | 13980 |
> || 16           | 15515                        | 9418        | 18601 |
> || 32           | 17699                        | 5421        | 20540 |
> || 64           | 20601                        | 4038        | 25591 |
> || 96           | 19177                        | 3891        | 26017 |
> We can clearly observe that the throughtput degrades as we increase the number of concurrent
requests, which led me to believe that there's severe context switching overhead and I could
indirectly confirm that suspicion with cs entry in vmstat output. branch-1.0 shows a much
higher number of context switches even with much lower throughput.
> Here are the observations:
> - WriteEntry in the writeQueue can only be removed by the very handler that put it, only
when it is at the front of the queue and marked complete.
> - Since a WriteEntry is marked complete after the wait-loop, only one entry can be removed
at a time.
> - This stringent condition causes O(N^2) context switches where n is the number of concurrent
handlers processing requests.
> So what I tried here is to mark WriteEntry complete before we go into wait-loop. With
the change, multiple WriteEntries can be shifted at a time without context switches. I changed
writeQueue to LinkedHashSet since fast containment check is needed as WriteEntry can be removed
by any handler.
> The numbers look good, it's virtually identical to pre-HBASE-8763 era.
> || Num threads || branch-1.0 with fix ||
> || 1            | 2459                 |
> || 2            | 4976                 |
> || 4            | 8033                 |
> || 8            | 12292                |
> || 16           | 15234                |
> || 32           | 16601                |
> || 64           | 19994                |
> || 96           | 20052                |
> So what do you think about it? Please let me know if I'm missing anything.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message