Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 21 Mar 2017 01:15:42 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13056977.1489753574000.73748.1490058942143@Atlassian.JIRA>
In-Reply-To: <JIRA.13056977.1489753574000@Atlassian.JIRA>
References: <JIRA.13056977.1489753574000@Atlassian.JIRA> <JIRA.13056977.1489753574222@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HBASE-17798) RpcServer.Listener.Reader can abort
 due to CancelledKeyException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Tue, 21 Mar 2017 01:15:48 -0000


     [ https://issues.apache.org/jira/browse/HBASE-17798?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu updated HBASE-17798:
---------------------------
    Attachment: 17798-master-v2.patch

> RpcServer.Listener.Reader can abort due to CancelledKeyException
> ----------------------------------------------------------------
>
>                 Key: HBASE-17798
>                 URL: https://issues.apache.org/jira/browse/HBASE-17798
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24
>            Reporter: Guangxu Cheng
>         Attachments: 17798-master-v2.patch, connections.png, HBASE-17798-=
0.98-v1.patch, HBASE-17798-0.98-v2.patch, HBASE-17798-branch-1-v1.patch, HB=
ASE-17798-branch-1-v2.patch, HBASE-17798-master-v1.patch, HBASE-17798-maste=
r-v2.patch
>
>
> In our production cluster(0.98), some of the requests were unacceptable b=
ecause RpcServer.Listener.Reader were aborted.
> getReader() will return the next reader to deal with request.
> The implementation of getReader() as below=EF=BC=9A
> {code:title=3DRpcServer.java|borderStyle=3Dsolid}
>     // The method that will return the next reader to work with
>     // Simplistic implementation of round robin for now
>     Reader getReader() {
>       currentReader =3D (currentReader + 1) % readers.length;
>       return readers[currentReader];
>     }
> {code}
> If one of the readers abort, then it will lead to fall on the reader's re=
quest will never be dealt with.
> Why does RpcServer.Listener.Reader abort?We add the debug log to get it.
> After a while, we got the following exception:
> {code}
> 2017-03-10 08:05:13,247 ERROR [RpcServer.reader=3D3,port=3D60020] ipc.Rpc=
Server: RpcServer.listener,port=3D60020: unexpectedly error in Reader(Throw=
able)
> java.nio.channels.CancelledKeyException
>         at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:=
73)
>         at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
>         at java.nio.channels.SelectionKey.isReadable(SelectionKey.java:28=
9)
>         at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoo=
p(RpcServer.java:592)
>         at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcS=
erver.java:566)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx=
ecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE=
xecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> {code}
> So, when deal with the request in reader, we should handle CanceledKeyExc=
eption.
> ----------
> versions 1.x and 2.0 will log and retrun when dealing with the Interrupte=
dException in Reader#doRunLoop after HBASE-10521. It will lead to the same =
problem.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)