tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Hayward" <john.ice.hayw...@gmail.com>
Subject Re: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang
Date Thu, 08 Feb 2007 00:00:25 GMT
We've seen errors where individual queries are improperly bounded or
unbounded allowing requests to attempt to retrieve unreasonable amounts of
data.  A single request like this can take an otherwise stable system and
throw it into all sorts of bad behaviors and error conditions.

To assist with this debugging you may try to run your jvm with gc logging
turned on, and log the http requests coming in.  If you then have an OOM
crash you can correlate the spike in memory allocation (on a timeline) with
certain requests that may have caused it.

you can use tools like gcviewer to post-process your gc log files to see
what was going on with your memory over time.

-John

On 2/7/07, Sharon French <sfrench@xtensible.com> wrote:
>
>
> > Uhhh - no.  The JVM will not return an OOME until the GC
> > is completely finished and the request still cannot be
> > satisfied.  Even with concurrent GC, failing allocation
> > requests are suspended until a full GC has completed.
> > The exception to the above is if a request is made that
> > could never be satisfied; that will be given an OOME
> > immediately.
>
> This is very helpful to know.  So if after receiving such OOMEs, I run
> jmap
> -heap and there is plenty of memory, I'm assuming that whomever was
> sucking
> up the memory has either given it back or died off - right?
>
> (The dying off would line up with fact that the number of processors
> tomcat
> says he's started is less than the number of threads I currently see
> running.)
>
> Now it's down to how do I catch the culprit?
>
> Since I don't see steady memory growth, I believe that there must be some
> error condition that is handled improperly that throws us into some sort
> of
> memory cosumption craziness.  Since by the time the symptom is present,
> there is plenty of memory available, I'm having trouble catching the case.
> Any suggestions?
>
> >> After an OOM is thrown, the JVM is basically completely hosed.
>
> > Definitely not true.  The JVM is still functional, although the app
> > (Tomcat, in this case) may not be able to proceed.
>
> This also lines up with my observations. The first several OOMEs don't
> seem
> to trip us up, however at some point we hit a threshold that does hose us.
>
> Given that after such an error occurs, we have plently of memory, and
> plenty
> of threads sitting in await(), any ideas on why the connections to apache
> end up in SYN_RECV? Am I right in thinking that this indicates that the
> tomcat listener thread has accepted the connection but then fails to pass
> it
> off to a processor thread to handle?  Or does the pass off happen but for
> some reason the socket is hosed?  Any thoughts here?
>
>
> > > If so, how do we combat this behavior?
> >
> > Fix resource leaks, allocate an appropriate amount of memory,
> > and limit resource use to avoid OOMs in the first place.
>
> > However, the 64m given to the PermGen is rather small for an app server.
>
> We don't seem to ever get to a usage of > 20M for PermGen, and the
> OutOfMemoryErrors always indicate "java heap space".  Is this error
> message
> reliable? Can I count on the fact that it's really heap space I'm out of
> (I've seen reports of this indicating other problems, but all seem to be
> pre-1.5 jvm).
>
> Thanks for the help thus far!
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>


-- 
John Hayward

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message