tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Eggers <its_toas...@yahoo.com>
Subject Re: Tomcat dies suddenly
Date Thu, 04 Feb 2010 04:46:29 GMT
Carl,

A couple of random thoughts . . .

I'm not familiar with the Slackware monitoring tools, but I am with the various tools that
come with Fedora / Redhat. One of the things that I've noticed with those GUI tools is that
they add cache and buffers to the free memory total.

Tools like top and vmstat should give a more complete picture of your memory. With vmstat
you can watch free, cache, buffers, and swap conveniently. With top, you can actually do a
command line monitor and watch a particular PID.

From the taroon-list: If you're running a 32 bit Linux and run out of low memory, it doesn't
matter how much high memory you have, the OOM killer will start killing processes off. Since
you're running a 64 bit Linux, this should not be the problem.

A discussion on stackoverflow.com may be more relevant to your situation. It turns out (according
to the discussion) that calling Runtime.getRuntime().exec() on a busy system can lead to transient
memory shortages which trigger the OOM killer.

If Runtime.getRuntime().exec() or similar calls do not exist in your application, then please
skip the following speculation. I've made some comments concerning host resolution at the
end of this message which might be helpful.

If Runtime.getRuntime().exec() is used, the scenario goes like this:

1. call Runtime.getRuntime().exec()
2. fork() gets called and makes a copy of the parent process
3. System runs a different process
   At this point you have two processes with largish memory requirements
   At this point the OOM killer may get triggered
4. exec() gets called on the child process and memory requirements go back down.

At least that's how I read the this reference:

http://stackoverflow.com/questions/209875/from-what-linux-kernel-libc-version-is-java-runtime-exec-safe-with-regards-to-m

Since processes that fork a lot of child processes are high on OOM killer's kill list, Tomcat
gets killed.

See for example: http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/

As to why it would happen on the newer production systems and not the older system, my only
idea concerns the version of the kernel you're using. Memory management has been significantly
reworked between the 2.4 and 2.6 kernels. If you use a 2.4 kernel on your older system, this
could explain some of the differences with memory allocation.

So, if Runtime.getRuntime().exec() is used, what are some possible solutions?

1. Reducing Xms, Xmx while adding physical memory

If you do this, then the fork() call without the exec() being called directly afterwards won't
be as expensive. Your application will be able to serve more clients without potentially triggering
the OOM killer.

Garbage collection may be an issue if this is done, so tuning with JMeter is probably a good
idea.

2. Create a lightweight process that forks what Runtime.getRuntime().exec() calls and communicate
with the process over sockets.

This is pretty unpleasant, but you might be able to treat this as a remote process server.
You could then end up using a custom object, JNDI lookups, and pooling, much like database
pooling.

As I've said, this is all based on an assumption that the application is requesting a transiently
large amount of memory caused by Runtime.getRuntime().exec() or other similar action. If this
is not the case, then the above arguments are null and void.

DNS Thoughts

As for the ideas concerning DNS - I've never seen DNS issues actually take down an environment.
However, I've seen orders of magnitude performance issues caused by poorly configured DNS
resolution and missing DNS entries.

One way to test DNS performance issues is to set up a client with a static IP address, but
don't put it in your local DNS. Then run JMeter on this client and stress your server. Finally,
add the client into DNS and stress your server with JMeter. If you notice a difference, then
there are some issues with how your server uses host resolution.

Make sure that nonexistent address resolution services (nisplus, nis, hesiod) are not listed
as sources on the host line in /etc/nsswitch.conf (or wherever Slackware puts it). At least
put a [NOTFOUND=return] entry after dns but before all the other services listed on the hosts:
line of the nsswitch.conf file.

So, here's a summary to all of this rambling:

1. Monitor memory with vmstat and top to get a better picture of the 
   system memory
2. If Runtime.getRuntime().exec() is used, then transient memory 
   allocations could trigger the OOM killer on a busy system
3. Make sure host resolution works properly, and turn it off in server.xml

OK, enough rambling - hope this is useful.

/mde/

--- On Wed, 2/3/10, Carl <carl@etrak-plus.com> wrote:

> From: Carl <carl@etrak-plus.com>
> Subject: Re: Tomcat dies suddenly
> To: "Tomcat Users List" <users@tomcat.apache.org>
> Date: Wednesday, February 3, 2010, 5:07 PM
> Chris,
> 
> Interesting idea.  I tried over the weekend to force
> that situation with JMeter hitting a simple jsp that did
> some data stuff and created a small display.  I pushed
> it to the point that there were entries in the log stating
> it was out of memory (when attempting to GC, I think) but it
> just slowed way down and never crashed.  I could see
> from VisualJVM that it had used the entire heap but, again,
> I could never get it to crash.
> 
> Strange because it doesn't have the classic signs (slowing
> down or throwing out of memory exceptions or freezing), it
> just disappears without any tracks.  I am certain there
> is a reason somewhere, I just haven't found it yet.
> 
> Thanks for your suggestions,
> 
> Carl



      


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message