tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suvendu Sekhar Mondal <suv3...@gmail.com>
Subject Re: Tomcat 7 giving java.lang.OutOfMemoryError: unable to create new native thread Exception in Catalina.out after upgrading to RHEL 7.4
Date Tue, 19 Sep 2017 13:53:23 GMT
Radhika,

On Tue, Sep 19, 2017 at 2:21 PM, Peddi, Radhika (Radhika)
<rpeddi@avaya.com> wrote:
> Hi,
>
> We have upgraded RHEL 7.2 to RHEL 7.4. After upgrade when we are running performance
testing of our application we are seeing below error in catalina.out.
>
> java.lang.OutOfMemoryError: unable to create new native thread Exception in Catalina.out
after upgrading to RHEL 7.4
>

This simply indicates that JVM was trying to create a new thread and
OS can't create any new thread simply because native memory space was
exhausted. This limit is very much platform dependent.

Is your app opening too many threads? On my Windows 10 laptop with
32GB RAM and JDK 1.8, I can open 530000+ threads before run into this
OOM problem. You can find it by monitoring Tomcat instance. So far I
ran into this problem one time - couple of years ago. We got away
after adding some memory and by reducing thread stack size.

Another thing can happen, you might have lost some OS setting(which
was used to bump up the default OS limit) while upgrading from one
version to another; which was preventing this issue from occurring so
far - just speculating. And this is not a Tomcat's problem. :)

> Attached is the thread dump.
>
> We are using Tomcat 7.0.69.
>
> As per REDHAT we have increased tomcat5 user which used to run Tomcat process in /etc/security/limits.d/20-nproc.conf
like below
>
> * soft nproc 4096
> root soft nproc unlimited
> tomcat5 soft nproc 8192
>
> All the processes are used by Tomcat. If we increase the value to 16K or 32K all threads
are consumed and they are in Waiting(Parking) state.
>
> "http-bio-8443-exec-56" #973 daemon prio=5 os_prio=0 tid=0x00007efc8c029800 nid=0x10e7b
waiting on condition [0x00007ef9bc7ee000]
>    java.lang.Thread.State: WAITING (parking)
>                 at sun.misc.Unsafe.park(Native Method)
>                 - parking to wait for  <0x00000003c454c008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>                 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>                 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>                 at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
>                 at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
>                 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>                 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>                 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>                 at java.lang.Thread.run(Thread.java:745)
>
> "http-bio-8443-exec-55" #972 daemon prio=5 os_prio=0 tid=0x00007efc0401e000 nid=0x10e7a
waiting on condition [0x00007ef9bc82f000]
>    java.lang.Thread.State: WAITING (parking)
>                 at sun.misc.Unsafe.park(Native Method)
>                 - parking to wait for  <0x00000003c454c008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>                 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>                 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>                 at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
>                 at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
>                 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>                 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>                 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>                 at java.lang.Thread.run(Thread.java:745)
>
> They are not getting release. After some time Tomcat getting stopped automatically. You
may need the output of below from our setup.
>
> [root@iqsvtk05<mailto:root@iqsvtk05> ~]# rpm -qif /opt/coreservices/tomcat-7.0.69/logs/
> Name : tomcat
> Version : 7.0.69
> Release : AV10
> Architecture: x86_64
> Install Date: Fri 18 Aug 2017 01:41:54 AM MDT
> Group : System Environment/Daemons
> Size : 8468465
> License : Apache License, Version 2.0
> Signature : (none)
> Source RPM : tomcat-7.0.69-AV10.src.rpm
> Build Date : Fri 24 Jun 2016 01:59:37 AM MDT
> Build Host : puiqx3650dev03.apac.avaya.com
> Relocations : /usr/share
> Packager : Avaya, Inc.
> Vendor : Apache Software Foundation
> URL : http://tomcat.apache.org/
> Summary : Core Services Apache Jakarta Servlet/JSP server
> Description : Servlet container for developing Web applications in Java. Used as basis
for the
> Reference Implementation of the Servlet and JSP specificiations. This RPM mirrors the
layout used by official Tomcat distributions on all other OSes and linux variants. If you
want a FHS-based package - be warned that tomcat may not work very well. (TODO: add more warnings
!). If you still want FHS-based package - please use jpackage.org or an RPM that is using
the same exact layout.
>
> Can you please help on this.
>
> Regards,
> Radhika

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message