Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E3901104D8 for ; Fri, 20 Dec 2013 20:52:11 +0000 (UTC) Received: (qmail 97547 invoked by uid 500); 20 Dec 2013 20:52:11 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 97510 invoked by uid 500); 20 Dec 2013 20:52:11 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 97501 invoked by uid 99); 20 Dec 2013 20:52:11 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Dec 2013 20:52:11 +0000 Date: Fri, 20 Dec 2013 20:52:11 +0000 (UTC) From: "stack (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-8558) Add timeout limit for HBaseClient dataOutputStream MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13854549#comment-13854549 ] stack commented on HBASE-8558: ------------------------------ We need this in 0.96+ too, right [~xieliang007]. Patch looks good to me. > Add timeout limit for HBaseClient dataOutputStream > -------------------------------------------------- > > Key: HBASE-8558 > URL: https://issues.apache.org/jira/browse/HBASE-8558 > Project: HBase > Issue Type: Bug > Components: Client > Affects Versions: 0.94.5, 0.94.14 > Reporter: wanbin > Assignee: Liang Xie > Attachments: HBASE-8558-0.94.txt > > > I run jstack at client host. The result is below. > "hbase-tablepool-60-thread-34" daemon prio=10 tid=0x00007f1e65a48000 nid=0x5173 runnable [0x00000000579cc000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked <0x0000000758cb0780> (a sun.nio.ch.Util$2) > - locked <0x0000000758cb0770> (a java.util.Collections$UnmodifiableSet) > - locked <0x0000000758cb0548> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336) > at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158) > at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:153) > at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:114) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > - locked <0x0000000754e978a0> (a java.io.BufferedOutputStream) > at java.io.DataOutputStream.flush(DataOutputStream.java:106) > at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:620) > - locked <0x0000000754e97880> (a java.io.DataOutputStream) > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975) > at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86) > at $Proxy13.multi(Unknown Source) > at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1395) > at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1393) > at org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:210) > at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1402) > at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1390) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > This thread have hung for one hours > Meanwhile other thread try to close connection > "IPC Client (1983049639) connection to dump002030.cm6.tbsite.net/10.246.2.30:30020 from admin" daemon prio=10 tid=0x00007f1e70674800 nid=0x3d76 waiting for monitor entry [0x000000004bc0f000] > java.lang.Thread.State: BLOCKED (on object monitor) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > - waiting to lock <0x0000000754e978a0> (a java.io.BufferedOutputStream) > at java.io.DataOutputStream.flush(DataOutputStream.java:106) > at java.io.FilterOutputStream.close(FilterOutputStream.java:140) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237) > at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:254) > at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.close(HBaseClient.java:715) > - locked <0x0000000754e7b818> (a org.apache.hadoop.hbase.ipc.HBaseClient$Connection) > at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:587) > dump002030.cm6.tbsite.net is dead regionserver. > I read hbase sourececode, discover connection.out doesn't set timeout > this.out = new DataOutputStream > (new BufferedOutputStream(NetUtils.getOutputStream(socket))); > I see this mean epoll_wait will block indefinitely. -- This message was sent by Atlassian JIRA (v6.1.4#6159)