Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (nike.apache.org: domain of billdsd@gmail.com designates
 209.85.192.180 as permitted sender)
Message-ID: <52A64675.1030600@gmail.com>
Date: Mon, 09 Dec 2013 14:38:45 -0800
From: Bill Davidson <billdsd@gmail.com>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:24.0) Gecko/20100101 Thunderbird/24.0
MIME-Version: 1.0
To: users@tomcat.apache.org
Subject: Notification strategy for OutOfMemoryError
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Last week, one of my servers got an OutOfMemoryError at approximately
1:21pm.

My monitoring software which does a heart beat check once per minute
did not notice until 3:01pm.  Heart beat kept working for over an hour
and a half.

During that time my high capacity high availablity 24/7 application was
getting occasional OutOfMemoryError's until memory got bad enough that
even the heart beat check servlet failed.  Apparently some things that
allocate large chunks of memory started failing first, but none of my
customers called to complain.  Smaller stuff continiued to work.  I didn't
know until my monitoring software sent me an email about the heart
beat failure.

That doesn't work for me.  I need to know sooner.

I thought of trying to handle it with error-page in web.xml. Apparently
that does not work.  I used java.lang.Throwable as the exception-type.
I was already using this for a number of common exceptions to send
me email.

I see the OutOfMemoryError's logged in my catalina.out

Is there some way that I can catch this so that I can send email
or something?  I need to know as soon as possible so that I can
attempt diagnosis and restart the server.  Google has not been helpful.
Everything says that you have to fix the memory leak.  Duh.  I know that.
We've fixed many over the years.  We haven't had one in nearly 2 years.
We thought we'd fixed them all.  We need to find out about them sooner
when they do happen.

This was on Tomcat 6.0.37 in production but we are about to upgrade to
7.0.47 and have been using that in development for a little over a month.
My testing has been in development (7.0.47) and I tested by explicitly
throwing OutOfMemoryError from a servlet.  Logged in catalina.out but
no other indication that it happened.  My error page does not get called.

     <error-page>
<exception-type>java.lang.Throwable</exception-type>
         <location>/jsp/error/Error.jsp</location>
     </error-page>

RedHat EL 5.9.  Java 7.0.25 64-bit.  Heap size 3GB.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org