hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11277) RPCServer threads can wedge under high load
Date Sat, 31 May 2014 01:21:01 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014448#comment-14014448
] 

Andrew Purtell commented on HBASE-11277:
----------------------------------------

A potential future improvement in this area is to track connection state with an enum and
switch to state specific handler blocks accordingly. I hacked around with that a bit today
but prefer to fix the issue at hand and maybe do that later.

> RPCServer threads can wedge under high load
> -------------------------------------------
>
>                 Key: HBASE-11277
>                 URL: https://issues.apache.org/jira/browse/HBASE-11277
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.96.2, 0.98.3
>            Reporter: Andrew Purtell
>            Assignee: Andrew Purtell
>            Priority: Critical
>             Fix For: 0.99.0, 0.96.3, 0.98.3
>
>         Attachments: HBASE-11277.patch
>
>
> This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load, RPCServer
threads can wedge, fail to make progess, and consume 100% CPU time on a core indefinitely.

> Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE threads
are mostly in EPollArrayWrapper.epollWait or FileDispatcherImpl.read0. The number of threads
found in FileDispatcherImpl.read0 correspond to the number of runaway threads expected based
on looking at 'top' output. These look like:
> {noformat}
> Thread 64758: (state = IN_NATIVE)
>  - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int) @bci=0 (Compiled
frame; information may be imprecise)
>  - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int) @bci=4, line=39
(Compiled frame)
>  - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor, java.nio.ByteBuffer,
long, sun.nio.ch.NativeDispatcher) @bci=114, line=223 (Compil
> ed frame)
>  - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher)
@bci=48, line=197 (Compiled frame)
>  - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379 (Compiled
frame)
>  - org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
>  - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess() @bci=509, line=1488
(Compiled frame)
>  - org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
@bci=23, line=790 (Compiled frame)
>  - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97, line=581
(Compiled frame)
>  - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1, line=556 (Interpreted
frame)
>  - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1145 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted
frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message