hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-3154) HBase RPC should support timeout
Date Thu, 24 Mar 2011 00:13:05 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010526#comment-13010526
] 

stack commented on HBASE-3154:
------------------------------

This is what it looks like on server-side:

{code}
2011-03-23 16:37:08,498 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call
multi(org.apache.hadoop.hbase.client.MultiAction@6db16bc) from 10.4.8.34:42287: output error
2011-03-23 16:37:08,498 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 61020
caught: java.nio.channels.ClosedChannelException
    at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
    at org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1387)
    at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1339)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)

2011-03-23 16:37:08,515 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call
multi(org.apache.hadoop.hbase.client.MultiAction@5754a4a7) from 10.4.8.34:42287: output error
2011-03-23 16:37:08,516 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 61020
caught: java.nio.channels.ClosedChannelException
    at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
    at org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1387)
    at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1339)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
{code}

On client side it looks like this:

{code}
11/03/23 17:03:16 DEBUG client.HConnectionManager$HConnectionImplementation: Failed all from
sv4borg230:61020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Call to sv4borg230/10.4.12.34:61020
failed on socket timeout exception: java.net.SocketTimeoutException: 1000 millis timeout while
waiting for channel to
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1158)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1246)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:822)
        at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:678)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:663)
        at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:316)
        at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:390)
        at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:155)
        at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:372)
        at com.yahoo.ycsb.ClientThread.run(Client.java:251)
Caused by: java.net.SocketTimeoutException: Call to sv4borg230/10.4.12.34:61020 failed on
socket timeout exception: java.net.SocketTimeoutException: 1000 millis timeout while waiting
for channel to be ready for read. ch : java.
        at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
        at $Proxy5.multi(Unknown Source)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4$1.call(HConnectionManager.java:1078)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4$1.call(HConnectionManager.java:1076)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1035)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1075)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1073)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: 1000 millis timeout while waiting for channel
to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.4.8.34:37568
remote=sv4borg230/10.4.12.34:61020]
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
        at java.io.FilterInputStream.read(FilterInputStream.java:116)
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:299)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:539)
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:477)
{code}



> HBase RPC should support timeout
> --------------------------------
>
>                 Key: HBASE-3154
>                 URL: https://issues.apache.org/jira/browse/HBASE-3154
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.90.0
>
>         Attachments: hbaseRpcTimeoutTrunk.patch
>
>
> We see a client stuck at an RPC to a zombie RS. HBase RPC should support timeout.
> "IPC Client (47) connection to msgstore179.ash2.facebook.com/10.138.89.182:60020 from
an unknown user" daemon prio=10 tid=0x00007f554c5ab800 nid=0x5df3 runnable [0x00000000590d4000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00007f5aa402a9e8> (a sun.nio.ch.Util$1)
> - locked <0x00007f5aa402aac0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00007f596a398408> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.FilterInputStream.read(FilterInputStream.java:116)
> at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:279)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> - locked <0x00007f5a1d7e7bd8> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readInt(DataInputStream.java:370)
> at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:514)
> at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:454)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message