hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "xufeng (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5673) The OOM problem of IPC client call cause all handle block
Date Thu, 29 Mar 2012 08:03:39 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241065#comment-13241065
] 

xufeng commented on HBASE-5673:
-------------------------------

I found this issue in my cluster.

1.I found any regionserver call not report to master because sockettimeout.
{noformat}
[2012-03-26 14:48:09,815] [INFO ] [regionserver20020] [org.apache.hadoop.hbase.regionserver.HRegionServer
1469] Attempting connect to Master server at DDB03:20000
[2012-03-26 14:49:09,818] [INFO ] [regionserver20020] [org.apache.hadoop.ipc.HbaseRPC 360]
Problem connecting to server: DDB03/192.168.28.53:20000
[2012-03-26 14:49:09,819] [WARN ] [regionserver20020] [org.apache.hadoop.hbase.regionserver.HRegionServer
1483] Unable to connect to master. Retrying. Error was:
java.net.SocketTimeoutException: Call to DDB03/192.168.28.53:20000 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel
to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.28.53:59520
remote=DDB03/192.168.28.53:20000]
{noformat}

2.through the jstack log of master,I found that one handle is waitting and others is blocked(waitForMeta).
{noformat}
。。。。。。。。。。。。
"IPC Server handler 90 on 20000" daemon prio=10 tid=0x00007f219c540000 nid=0x4c3f in Object.wait()
[0x00007f21963a7000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:757)
	。。。。。。。。。

"IPC Server handler 87 on 20000" daemon prio=10 tid=0x00007f219c53a000 nid=0x4c37 waiting
for monitor entry [0x00007f21966aa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:397)
	- waiting to lock <0x0000000612486960> (a java.util.concurrent.atomic.AtomicBoolean)
	at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:437)
。。。。。。。。。。。
{noformat}

3.I also ensure that the waitting handle cause the others blocked,the waitting handle is waitting
for the call to complete.

4.But the unable to create new native thread” happened, the IOException can not caught it.
{noformat}
protected synchronized void setupIOstreams() throws IOException {
。。。。
        start();
      } catch (IOException e) {
        markClosed(e);
        close();

        throw e;
      }
。。。。。
{noformat}


5.thus the call will be lost in call queue and never to complete.
{noformat}
public Writable call(......)
{
......
    synchronized (call) {
      while (!call.done) {
        try {
          call.wait();                           // wait for the result
        } catch (InterruptedException ignored) {
          // save the fact that we were interrupted
          interrupted = true;
        }
      }
......
}

{noformat}
                
> The OOM problem of IPC client call  cause all handle block
> ----------------------------------------------------------
>
>                 Key: HBASE-5673
>                 URL: https://issues.apache.org/jira/browse/HBASE-5673
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.90.6
>         Environment: 0.90.6
>            Reporter: xufeng
>            Assignee: xufeng
>
> if HBaseClient meet "unable to create new native thread" exception, the call will never
complete because it be lost in calls queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message