Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of tianq01@gmail.com designates
 209.85.216.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <53C3BCAC.3050604@gmail.com>
References: 
 <CADJ9AzOKn=zoiUMzEZ_9zim4WSXp8n221NOVKu-Sst=JxU_AYg@mail.gmail.com>
	<CADJ9AzPVQJ_Z=+KjXWiCRTnVEF5e0-GO5QoYPk9UZdC0B6wTkA@mail.gmail.com>
	<53C1FF5D.9060408@gmail.com>
	<1405282709523-4061293.post@n3.nabble.com>
	<53C33165.6010400@gmail.com>
	<CAMZUsP7VMASOgEMoS4H1nLf8GfZGbXeyMVZK+TYTGSw6nzmqHw@mail.gmail.com>
	<53C3BCAC.3050604@gmail.com>
Date: Tue, 15 Jul 2014 10:24:48 +0800
Message-ID: 
 <CAMZUsP5Nd9Emub+yRRK1JAVqJD_kERqY-LKx+8QVrojnFHwwqA@mail.gmail.com>
Subject: Re: hbase region servers refuse connection
From: Qiang Tian <tianq01@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c2cfc897c0e104fe321c72

--001a11c2cfc897c0e104fe321c72
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

not sure if you could also try using different connections for your client
program. see the related issue HBASE-11306. I guess sharing connection  is
a contributing factor for this issue.


On Mon, Jul 14, 2014 at 7:19 PM, Rural Hunter <ruralhunter@gmail.com> wrote=
:

> Hi Tian Qiang,
>
> Good to hear that. It seems the jira is fixed. I will seek time to build
> the latest 0.96 branch and test it.
>
> =E4=BA=8E 2014/7/14 18:06, Qiang Tian =E5=86=99=E9=81=93:
>
>> Hi YuMing, :)
>> yes. several iterations of jstack on the problem regionserver could help
>> identify the problem
>>
>> Rural,
>> you probably hit hbase11277(and probably YuMin as well) - the reader 14
>> loops again and again in
>> below stack(high cpu usage) and listener 12 is blocked and cannot
>> accept new connections.
>>
>>
>>
>>     1. Thread 12 (RpcServer.listener,port=3D60020):
>>     2.   State: BLOCKED
>>     3.   Blocked count: 123264191
>>     4.   Waited count: 0
>>     5.   Blocked on
>>     org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@77f87716
>>     6.   Blocked by 14 (RpcServer.reader=3D1,port=3D60020)
>>     7.   Stack:
>>     8.
>>     org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.
>> registerChannel(RpcServer.java:598)
>>     9.
>>     org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(
>> RpcServer.java:755)
>>     10.
>>     org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(
>> RpcServer.java:673)
>>     11. Thread 24 (RpcServer.responder):
>>
>>
>>
>>     1. Thread 14 (RpcServer.reader=3D1,port=3D60020):
>>     2.   State: RUNNABLE
>>     3.   Blocked count: 12510492
>>     4.   Waited count: 12826560
>>     5.   Stack:
>>     6.     sun.nio.ch.FileDispatcher.read0(Native Method)
>>     7.     sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>     8.     sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)
>>     9.     sun.nio.ch.IOUtil.read(IOUtil.java:224)
>>     10.     sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254=
)
>>     11.
>>     org.apache.hadoop.hbase.ipc.RpcServer.channelIO(RpcServer.java:2438)
>>     12.
>>     org.apache.hadoop.hbase.ipc.RpcServer.channelRead(
>> RpcServer.java:2404)
>>     13.
>>     org.apache.hadoop.hbase.ipc.RpcServer$Connection.
>> readAndProcess(RpcServer.java:1498)
>>     14.
>>     org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(
>> RpcServer.java:780)
>>     15.
>>     org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.
>> doRunLoop(RpcServer.java:568)
>>     16.
>>     org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(
>> RpcServer.java:543)
>>     17.
>>     java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1146)
>>     18.
>>     java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>     19.     java.lang.Thread.run(Thread.java:701)
>>     20. Thread 13 (RpcServer.reader=3D0,port=3D60020):
>>     21.
>>
>>
>>
>>     1. 2014-07-10 14:13:49,614 WARN  [RpcServer.reader=3D7,port=3D60020]
>>
>>     ipc.RpcServer: RpcServer.listener,port=3D60020: count of bytes read:=
 0
>>     2. java.io.IOException: Connection reset by peer
>>     3.         at sun.nio.ch.FileDispatcher.read0(Native Method)
>>     4.         at sun.nio.ch.SocketDispatcher.
>> read(SocketDispatcher.java:39)
>>     5.         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251=
)
>>     6.         at sun.nio.ch.IOUtil.read(IOUtil.java:224)
>>     7.         at
>>     sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
>>     8.         at
>>     org.apache.hadoop.hbase.ipc.RpcServer.channelRead(
>> RpcServer.java:2404)
>>     9.         at
>>     org.apache.hadoop.hbase.ipc.RpcServer$Connection.
>> readAndProcess(RpcServer.java:1425)
>>     10.         at
>>     org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(
>> RpcServer.java:780)
>>     11.         at
>>     org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.
>> doRunLoop(RpcServer.java:568)
>>     12.         at
>>     org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(
>> RpcServer.java:543)
>>     13.         at
>>     java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1146)
>>     14.         at
>>     java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>     15.         at java.lang.Thread.run(Thread.java:701)
>>
>>
>>
>>
>

--001a11c2cfc897c0e104fe321c72--