From dev-return-196232-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Mon Jan 28 10:50:23 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id D41A718060E for ; Mon, 28 Jan 2019 10:50:22 +0100 (CET) Received: (qmail 34509 invoked by uid 500); 28 Jan 2019 09:50:21 -0000 Mailing-List: contact dev-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Developers List" Delivered-To: mailing list dev@tomcat.apache.org Received: (qmail 34499 invoked by uid 99); 28 Jan 2019 09:50:21 -0000 Received: from mail-relay.apache.org (HELO mailrelay2-lw-us.apache.org) (207.244.88.137) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jan 2019 09:50:21 +0000 Received: from mail-io1-f47.google.com (mail-io1-f47.google.com [209.85.166.47]) by mailrelay2-lw-us.apache.org (ASF Mail Server at mailrelay2-lw-us.apache.org) with ESMTPSA id 9DBDE2DAB for ; Mon, 28 Jan 2019 09:50:20 +0000 (UTC) Received: by mail-io1-f47.google.com with SMTP id x6so12910412ioa.9 for ; Mon, 28 Jan 2019 01:50:20 -0800 (PST) X-Gm-Message-State: AHQUAuY+PdLxxDXZd7n/rHkgtDqgCj29HRL6nxJr4dpEv51SRGOkydeL ynazOwBqigPTfI4gU3BHcIinWM0umTQSjR824v0= X-Google-Smtp-Source: ALg8bN6suyl5XFhQQMXFq5geRluKGqOiN8XArNeGndDqkz4emyEjQjGchDMR0PrzNZenSZhsBdjXLiApmlO5ul5uUnU= X-Received: by 2002:a5e:d617:: with SMTP id w23mr10998586iom.75.1548669020109; Mon, 28 Jan 2019 01:50:20 -0800 (PST) MIME-Version: 1.0 References: <65cf4f1d-62e5-dfac-dfe5-8948e1a0161b@kippdata.de> In-Reply-To: <65cf4f1d-62e5-dfac-dfe5-8948e1a0161b@kippdata.de> From: =?UTF-8?Q?R=C3=A9my_Maucherat?= Date: Mon, 28 Jan 2019 10:50:08 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: TC trunk on SLES 11 with APR and Java 11: to many threads during TestDefaultServletEncodingWith*Bom To: Tomcat Developers List Content-Type: multipart/alternative; boundary="000000000000d184fc0580819a87" --000000000000d184fc0580819a87 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Jan 27, 2019 at 2:18 AM Rainer Jung wrote= : > When running the unit tests for trunk using Java 11 and APR connector I > run out of memory, first in threads, later in heap. The prpoblematic > tests are TestDefaultServletEncodingWithBom and > TestDefaultServletEncodingWithoutBom. > > A thread dumps shows 540 sendfile threads like the following: > > [junit] "http-apr-127.0.0.1-auto-540-Sendfile" #8113 daemon prio=3D5 > There should be one sendfile thread (at most) per APR connector. 540 is only a static counter to avoid running into a connector name conflict when the auto port is used. The sendfile thread is normally started and stopped with the connector. There's probably another cause for this apparent resource leak (I don't get the error on my OpenJDK 11). The counter is a great indication of the lifecycle of the JVM. R=C3=A9my > os_prio=3D0 cpu=3D0.34ms elapsed=3D8031.17s tid=3D0x00007fd4b8171800 nid= =3D0x1249 > in Object.wait() [0x00007fd497313000] > [junit] java.lang.Thread.State: WAITING (on object monitor) > [junit] at java.lang.Object.wait(java.base@11.0.2/Native Method) > [junit] - waiting on > [junit] at java.lang.Object.wait(java.base@11.0.2 > /Object.java:328) > [junit] at > org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116= ) > [junit] - waiting to re-lock in wait() <0x00000000f7b38278> (a > org.apache.tomcat.util.net.AprEndpoint$Sendfile) > [junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834= ) > [junit] > > Is it expected, that those tests use that many threads in parallel? > > The later heap memory OOME is probably also due to resource constraints > on that VM. The memory when the OOME happens is: > > [junit] Heap > [junit] def new generation total 76800K, used 67531K > [0x00000000f0600000, 0x00000000f5950000, 0x00000000f5950000) > [junit] eden space 68288K, 98% used [0x00000000f0600000, > 0x00000000f47f2e38, 0x00000000f48b0000) > [junit] from space 8512K, 0% used [0x00000000f48b0000, > 0x00000000f48b0000, 0x00000000f5100000) > [junit] to space 8512K, 0% used [0x00000000f5100000, > 0x00000000f5100000, 0x00000000f5950000) > [junit] tenured generation total 170688K, used 170687K > [0x00000000f5950000, 0x0000000100000000, 0x0000000100000000) > [junit] the space 170688K, 99% used [0x00000000f5950000, > 0x00000000ffffffd0, 0x0000000100000000, 0x0000000100000000) > [junit] Metaspace used 25248K, capacity 25384K, committed > 26624K, reserved 26624K > > In addition I see native errors (tcnative 1.2.21): > > [junit] Exception in thread "Catalina-utility-1" > java.lang.OutOfMemoryError: unable to create native thread: possibly out > of memory or process/resource limits reached > [junit] at java.base/java.lang.Thread.start0(Native Method) > [junit] [156.313s][warning][os,thread] Failed to start thread - > pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, > guardsize: 0k, detached. > [junit] [156.314s][warning][os,thread] Failed to start thread - > pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, > guardsize: 0k, detached. > [junit] at java.base/java.lang.Thread.start(Thread.java:803) > [junit] at > > java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExe= cutor.java:937) > [junit] at > > java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(Threa= dPoolExecutor.java:1005) > [junit] at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExe= cutor.java:1142) > [junit] at > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolEx= ecutor.java:628) > [junit] at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread= .java:61) > [junit] at java.base/java.lang.Thread.run(Thread.java:834) > > an then later after many of those: > > [junit] 26-Jan-2019 23:34:12.475 SEVERE [main] > org.apache.catalina.core.ContainerBase.startInternal A child container > failed during start > [junit] java.util.concurrent.ExecutionException: > java.lang.OutOfMemoryError: Java heap space > [junit] at > java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) > [junit] at > java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) > [junit] at > > org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:9= 09) > [junit] at > > org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java= :262) > [junit] at > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) > [junit] at > > org.apache.catalina.core.StandardService.startInternal(StandardService.ja= va:423) > [junit] at > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) > [junit] at > > org.apache.catalina.core.StandardServer.startInternal(StandardServer.java= :928) > [junit] at > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) > [junit] at > org.apache.catalina.startup.Tomcat.start(Tomcat.java:455) > [junit] at > > org.apache.catalina.startup.TomcatBaseTest$TomcatWithFastSessionIDs.start= (TomcatBaseTest.java:845) > > I do not know, whether the pthread errors happen after to many threads > were created or are the reson for the many threads to get created. > > Regards, > > Rainer > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org > For additional commands, e-mail: dev-help@tomcat.apache.org > > --000000000000d184fc0580819a87--