hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sunweiwei" <su...@asiainfo-linkage.com>
Subject 答复: regionserver 60020 timeout
Date Fri, 06 Jun 2014 23:38:41 GMT
Hi 

the regionserver jstack log is   http://paste2.org/2wJc5CJx
the master jstack log is   http://paste2.org/t4KLpbXw.   Maybe the regionserver is hbase:namespace
server.
the master dump log (from the 60010 web) is http://paste2.org/vB2UVPyO


I saw the regionserver's Requests Per Second is 0 at that time , 
other regionserver's Requests Per Second is about 10000~50000,  and some clients failed. 
I'm not sure about loading at that time.  regionserver's loading usually is not high.

I restart the regionserver and master then hbase back to normal.

Thanks
-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2014年6月7日 0:16
收件人: Hbase-User
主题: Re: regionserver 60020 timeout

On Thu, Jun 5, 2014 at 11:35 PM, sunweiwei <sunww@asiainfo-linkage.com>
wrote:

> Hi
>
> I find some 60000 millis timeout in master log, when master connect
> regionserver's 60020 port. Log like this:
>
>
Could we see the full stack trace?  It would help confirm Qiang's
prognosis.  Do you have monitoring of this server setup?  What did it show
for loading, handler-count, at this time?

Thanks,
St.Ack



> because java.net.SocketTimeoutException: 60000 millis timeout while waiting
> for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/132.228.248.61:34835
> remote=dsjhd074/132.228.248.84:60020]
>
> at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1475)
>
>          at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1450)
>
>          at
>
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1650
> )
>
>          at
>
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callB
> lockingMethod(RpcClient.java:1708)
>
>          at
>
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$Blocking
> Stub.openRegion(AdminProtos.java:20595)
>
>          at
>
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.ja
> va:628)
>
>          at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
> va:1901)
>
>          at
>
> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManage
> r.java:106)
>
>          at
>
> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager
> .java:660)
>
>          at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>
>          at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>          at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>          at java.lang.Thread.run(Thread.java:662)
>
>
>
> And I find listener,port 60020 is waiting to lock <0x000000041ae290d8>  in
> regionserver's jstack log. Is that ok? Thanks
>
> "RpcServer.reader=4,port=60020" daemon prio=10 tid=0x00007fade53c0000
> nid=0x22d8c runnable [0x00007fadb8b32000]
>
>    java.lang.Thread.State: RUNNABLE
>
>         at sun.nio.ch.FileDispatcher.read0(Native Method)
>
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
>
>         at sun.nio.ch.IOUtil.read(IOUtil.java:171)
>
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
>
>         - locked <0x000000041b1c78c0> (a java.lang.Object)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcServer.channelIO(RpcServer.java:2402)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcServer.channelRead(RpcServer.java:2368)
>
>         at
>
> org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.ja
> va:1476)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:770)
>
>         at
>
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.ja
> va:563)
>
>         - locked <0x000000041ae290d8> (a
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
>
>         at
>
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538
> )
>
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
> "RpcServer.listener,port=60020" daemon prio=10 tid=0x00007fadd4426800
> nid=0x22db3 waiting for monitor entry [0x00007fadb658a000]
>
>    java.lang.Thread.State: BLOCKED (on object monitor)
>
>         at
>
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcSer
> ver.java:593)
>
>         - waiting to lock <0x000000041ae290d8> (a
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:745)
>
>         at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:668)
>
>


Mime
View raw message