tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <rainer.j...@kippdata.de>
Subject Re: [Tomcat] [daemon] java.lang.OutOfMemoryError unable to create new native thread
Date Fri, 06 Feb 2009 16:25:48 GMT
On 03.02.2009 16:31, Steve Cohen wrote:
> We have an application that runs under Tomcat under RHEL 5.0  and is
> launched by a jsvc daemon.
> It chugs along seemingly fine on several servers, yet yesterday crashed
> on one of them with the above exception seemingly without experiencing
> any kind of abnormal load. I am trying to get a handle on this.
> System is
>
> $ uname -a
> Linux 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:29:46 EST 2008 i686 i686
> i386 GNU/Linux
>
> Java version:
> java -version
> java version "1.5.0_16"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
> Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing)
>
> no special JVM memory settings are used when launching tomcat, just the
> default memory allocations. I have read in some places that this
> particular form of OutOfMemoryError is actually made worse by increasing
> the memory size.
>
> I need to deploy some thread and memory monitoring setup to get a handle
> on this.

I wouldn't put so much work into making X11 work.

You need to find out your memory situation and your thread situation.

The error says: when trying to start a new thread, native memory 
allocation failed and thus the thread could not be created.

So the reason is, that you ran out of native memory in the process. The 
cure is either to reduce the amount of memory needed or to reduce the 
number of threads needed.

Concerning Memory:

Basically your process uses Java memory (Heap and Permanent generation) 
and native memory (thread stacks and other native memory).

If you are using a 32Bit system or VM, the address space is 4GB per 
process, but that space is divided in some part available for the 
process, and some part used by the kernel for data associated with the 
process. The part actually available for the process depends as far as i 
know on kernel settings and can be either 1, 2, or 3 GB (I would expect 
2GB).

Heap, Perm and native memory has to fit into this. I'm not quite sure 
about the thread stacks, but I assume those must fit into the 1/2/3 GB too.

Process memory can be inspected using the proc file system. The file 
/proc/MYPID/maps (MYPID is the process id of your Tomcat process) 
contains a table of memory areas used by the process. The hex numbers at 
the beginning of each line shows starting and ending addresses, the 
differences are the sizes of the areas.

Check whether you can verify, that the process is close to one of those 
magic limits.

Then check, how big your heap and your permanent generation is. You can 
do this with jstat. I would sugest you find a good sizing for your app 
and don't proceed with automatic 8and thus unknown) sizing.

If total memory is close to the limit and heap+perm is close to the 
total memory, you know that your java memory is to big for 32bits.

If total memory is close to the limit and heap+perm is much lower, then 
there's something which consumes large native memory. It could be 
threads (see next) or some embedded native component, e.g. some library 
included via JNI.

Threads:

Use "kill -QUIT MYPID" with MYPID the process id of your Tomcat. It will 
write a thread dump (not: memory dump) to catalina.out. This dump 
contains all Java threads of your process and the stacks they are in. 
See how many these are and to which components they belong. With a 
thread dump you can find out, whether you have extremely many Java 
threads. You need to be the owner of the Tomcat process or root to be 
able to send a signal to the process. You need to be able to read 
catalina.out to analyse the output.

Use the "-L" flag of "ps" to look at all threads of the Tomcat process. 
Check whether there are many more threads in ps, than there are in the 
Java thread dump. If so, you have some native component inside, that 
creates additional threads. In this case use gstack, to check, what 
those are doing (to get an idea, what they are used for).

If you can verify that you run our of native memory, but you can reduce 
neither memory size not thread count, you can start playing around with 
the thread stack size using -Xss. But this will be tedious.

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message