tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: Notification strategy for OutOfMemoryError
Date Wed, 11 Dec 2013 15:14:58 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Bill,

On 12/9/13, 8:20 PM, Bill Davidson wrote:
> On 12/9/2013 3:12 PM, Christopher Schultz wrote:
>> 
>> Was it a transient error, or a chronic condition? A single thread
>> can, for instance, spew objects into its stack or eden space
>> exhausting memory but, when that thread hits the OOME, all those
>> objects are freed which basically recovers from the situation.
>> 
>> If, instead, you fill-up some shared cache, buffer, etc. and NO 
>> threads can get more memory, then you're basically toast.
>> 
>> Which of the above was it?
> 
> It looked more like the first one though we still haven't tracked
> down the cause.  We had several dozen threads running at the time.
> That's common for us.  It's not that unusual for us to have a
> couple of hundred users with active sessions per server at any
> given time.
> 
>> There are a bunch of things you can try to do. They all have
>> their caveats, failure scenarios, and inefficacies.
>> 
>> 1. Use -XX:OnOutOfMemoryError="cmd args;cmd args"
>> 
>> Rig this to email you, register a passive-check data point with
>> your monitoring server, etc. Just remember that OOMEs happen for
>> a number of reasons. You could have run out of file handles or
>> you could have run out of heap space.
> 
> That looks interesting.  It wouldn't tell me about the error but
> at least I'd know that there was an OOME.  Better than nothing and
> I can go check catalina.out.  Of course, I still have the problem
> that threads silently fail and show my users not so much as an
> error message.
> 
>> 2. Use JMX monitoring, set java.lang:MemoryPool/[heap 
>> space]/UsageThreshold to whatever byte value you want to set as
>> your limit. Then, check java.lang:MemoryPool/[heap 
>> space]/UsageThresholdExceeded to see if it is true. If so, your
>> usage threshold has been exceeded.
>> 
>> Note that this is not proof-positive than an OOME occurred. It's
>> also tough to tell what value to use for the threshold. You can't
>> really set it to MaxHeap - 1 byte, because you'll never get that
>> value in practice. If you set it too low, you'll get warnings all
>> the time when your heap usage rises in the normal course of
>> business.
> 
> I'm less enthused about that one.
> 
>> 3. catch IOException in a filter and set an application
>> attribute. Check this attribute from your monitor.
>> 
>> I've been considering doing this, because I can rig it so that
>> the error handler does not actually require any memory to run.
>> The problem is that sometimes OOMEs interrupt one thread and not
>> another. You may not catch the OOME in that thread -- it may
>> happen in a background thread that does not go through the
>> filter.
> 
> I'm not sure I understand this one.  How does an IOException relate
> to an OOME?

Sorry, I meant of course OutOfMemoryError. Just make sure you use as
little memory as possible during the exception handler or it will fail
itself and possibly mask the original problem.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJSqIFyAAoJEBzwKT+lPKRYKxoP/iDp+6bzkKN3KgqsMziO2fSl
/NTbWIz1F9jzBAkFpRs+MQqwT3ZzUbE0QPbTnxPh3ShS+iD+UobtvdS2mn6YLNX9
qcWislmZKYMSFK/idc9JgBZE6XDRPG6bAo/X2lEfV9rURHVaA10QCgt7xDXdfN9b
5Ggs0ZfA5v2VQNIZyDFIukZzswFfA/VVb42vAR/wkTuBVVT/opZliv19gHc6y9D9
harx4z5cfEFQMq8YLdrQJIyPUXeoRoS9Um9oujS2FCWDEa0kni5Fn2nFh7beHHzi
CMjncYAQ57pm59LwWp0PzRsZmPsr/UmgwiM95yt+c6cqxZKCr3d9xqqAMaqC33Fj
sK8Hz/JzFkkfAK5xRHmdSqP+svpFpAvWJD75LOr4XvEOmEM1hCe4hkbzsa6wDJ5y
f7EHAVbCjbgqtE7Ic3G5Gxz7KlarlWn7QIvJHJfy1kD9KbSVJtph5O6d6fQkxaQG
vxeL9LqnMmPq8YmAlvuK/V2uUAmHNS4TVYKYTLnx/Z9kuiozhQSGoJAWP36KpM+3
kXZi9/9/L5SIry+R+PABj8UIKktIZ9ZJGZzV9uT8SiiEdkADz8kgcRWRUtdk3eDD
T9Dzt2Tr9SV+HKUVkE3MTpqpAO6NVroOZ57ij6diiD7WXEMb9WeZcUt7IxxNOf+A
nDT5bAeuQLHwAZTyQN74
=N4NL
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message