hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Performance degradation between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)
Date Tue, 08 Sep 2015 15:05:46 GMT
In HRegion#increment(), we lock the row (not region):

    try {
      rowLock = getRowLock(row);

Can you pastebin the complete stack trace ?

Thanks

On Tue, Sep 8, 2015 at 2:01 AM, 鈴木俊裕 <brfrn169@gmail.com> wrote:

> Hi,
>
> We upgraded our cluster from CDH5.3.1(HBase0.98.6) to CDH5.4.5(HBase1.0.0)
> and we experience slowdown in increment operation.
>
> Here's an extract from thread dump of the RegionServer of our cluster:
>
> Thread 68 (RW.default.writeRpcServer.handler=15,queue=5,port=60020):
>   State: BLOCKED
>   Blocked count: 21689888
>   Waited count: 39828360
>   Blocked on java.util.LinkedList@3474e4b2
>   Blocked by 63 (RW.default.writeRpcServer.handler=10,queue=0,port=60020)
>   Stack:
>     java.lang.Object.wait(Native Method)
>
>
> org.apache.hadoop.hbase.regionserver.MultiVersionConsistencyControl.waitForPreviousTransactionsComplete(MultiVersionConsistencyControl.java:224)
>
>
> org.apache.hadoop.hbase.regionserver.MultiVersionConsistencyControl.waitForPreviousTransactionsComplete(MultiVersionConsistencyControl.java:203)
>
> org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:6712)
>
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:501)
>
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:570)
>
>
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1901)
>
>
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31451)
>     org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
>     org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
>
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
>     org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>     java.lang.Thread.run(Thread.java:745)
>
> There are many similar threads in the thread dump.
>
> I read the source code and I think this is caused by changes of
> MultiVersionConsistencyControl.
> A region lock (not a row lock) seems to occur in
> waitForPreviousTransactionsComplete().
>
>
> Also we wrote performance test code for increment operation that included
> 100 threads and ran it in local mode.
>
> The result is shown below:
>
> CDH5.3.1(HBase0.98.6)
> Throughput(op/s): 12757, Latency(ms): 7.975072509210629
>
> CDH5.4.5(HBase1.0.0)
> Throughput(op/s): 2027, Latency(ms): 49.11840157868772
>
>
> Thanks,
>
> Toshihiro Suzuki
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message