hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chia-Ping Tsai (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-19204) branch-1.2 times out and is taking 6-7 hours to complete
Date Sat, 18 Nov 2017 11:56:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258038#comment-16258038
] 

Chia-Ping Tsai edited comment on HBASE-19204 at 11/18/17 11:55 AM:
-------------------------------------------------------------------

It is blocked when closing {{HttpServer2}}. Zoom in on the {{HttpServer2}}
{code}
  @Test
  public void testHttpServerHang() throws Exception {
    for (int i = 0; i != 500; ++i) {
      HttpServer2 server = null;
      try {
        Configuration conf = new Configuration();
        conf.setInt("hadoop.http.max.threads", 10);
        server = createTestServer(conf);
        server.addServlet("echo", "/echo", TestHttpServer.EchoServlet.class);
        server.addServlet("echomap", "/echomap", TestHttpServer.EchoMapServlet.class);
        server.addServlet("htmlcontent", "/htmlcontent", TestHttpServer.HtmlContentServlet.class);
        server.addServlet("longheader", "/longheader", TestHttpServer.LongHeaderServlet.class);
        server.addJerseyResourcePackage(
            JerseyResource.class.getPackage().getName(), "/jersey/*");
        server.start();
      } finally {
        if (server != null) {
          server.stop();
        }
      }
    }
  }
{code}
And then i got the following stack dump.
{quote}
"2020688994@qtp-1069352689-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39279"
daemon prio=10 tid=0x00007fbd8c7a4000 nid=0x1c3e waiting for monitor entry [0x00007fbd740e1000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:464)
        - waiting to lock <0x00000000ec24c798> (a org.mortbay.io.nio.SelectorManager$SelectSet)
        at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
        at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)


"main" prio=10 tid=0x00007fbd8c009000 nid=0x1bdc runnable [0x00007fbd93ad5000]
   java.lang.Thread.State: RUNNABLE
        at org.mortbay.io.nio.SelectorManager$SelectSet.stop(SelectorManager.java:878)
        - locked <0x00000000ec24c798> (a org.mortbay.io.nio.SelectorManager$SelectSet)
        at org.mortbay.io.nio.SelectorManager.doStop(SelectorManager.java:240)
        at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76)
        - locked <0x00000000ec07da28> (a java.lang.Object)
        at org.mortbay.jetty.nio.SelectChannelConnector.close(SelectChannelConnector.java:136)
{quote}
And then i noticed the same observation is already in the HBASE-14430 and HBASE-14505. [~stack]
Do you still look for the cryptic JVM version? :)


was (Author: chia7712):
It is blocked when closing {{HttpServer2}}. Zoom in on the {{HttpServer2}}
{code}
  @Test
  public void testHttpServerHang() throws Exception {
    for (int i = 0; i != 500; ++i) {
      HttpServer2 server = null;
      try {
        Configuration conf = new Configuration();
        conf.setInt("hadoop.http.max.threads", 10);
        server = createTestServer(conf);
        server.addServlet("echo", "/echo", TestHttpServer.EchoServlet.class);
        server.addServlet("echomap", "/echomap", TestHttpServer.EchoMapServlet.class);
        server.addServlet("htmlcontent", "/htmlcontent", TestHttpServer.HtmlContentServlet.class);
        server.addServlet("longheader", "/longheader", TestHttpServer.LongHeaderServlet.class);
        server.addJerseyResourcePackage(
            JerseyResource.class.getPackage().getName(), "/jersey/*");
        server.start();
      } finally {
        if (server != null) {
          server.stop();
        }
      }
    }
  }
{code}
And then i got the following stack dump.
{quote}
"2020688994@qtp-1069352689-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39279"
daemon prio=10 tid=0x00007fbd8c7a4000 nid=0x1c3e waiting for monitor entry [0x00007fbd740e1000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:464)
        - waiting to lock <0x00000000ec24c798> (a org.mortbay.io.nio.SelectorManager$SelectSet)
        at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
        at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)


"main" prio=10 tid=0x00007fbd8c009000 nid=0x1bdc runnable [0x00007fbd93ad5000]
   java.lang.Thread.State: RUNNABLE
        at org.mortbay.io.nio.SelectorManager$SelectSet.stop(SelectorManager.java:878)
        - locked <0x00000000ec24c798> (a org.mortbay.io.nio.SelectorManager$SelectSet)
        at org.mortbay.io.nio.SelectorManager.doStop(SelectorManager.java:240)
        at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76)
        - locked <0x00000000ec07da28> (a java.lang.Object)
        at org.mortbay.jetty.nio.SelectChannelConnector.close(SelectChannelConnector.java:136)
{quote}
And then i noticed the same observation is already in the HBASE-14430 and HBASE-14505. [~stack]
Are you still look for the cryptic JVM version? :)

> branch-1.2 times out and is taking 6-7 hours to complete
> --------------------------------------------------------
>
>                 Key: HBASE-19204
>                 URL: https://issues.apache.org/jira/browse/HBASE-19204
>             Project: HBase
>          Issue Type: Umbrella
>          Components: test
>            Reporter: stack
>
> Sean has been looking at tooling and infra. This Umbrellas is about looking at actual
tests. For example, running locally on dedicated machine I picked a random test, TestPerColumnFamilyFlush.
In my test run, it wrote 16M lines. It seems to be having zk issues but it is catching interrupts
and ignoring them ([~carp84] fixed this in later versions over in HBASE-18441).
> Let me try and do some fixup under this umbrella so we can get a 1.2.7 out the door.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message