tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Date Sun, 14 Apr 2013 22:51:37 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Howard,

On 4/11/13 10:38 PM, Howard W. Smith, Jr. wrote:
> On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz < 
> chris@christopherschultz.net> wrote:
> 
>> Your heap settings should be tailored to your environment and
>> usage scenarios.
> 
> Interesting. I suppose 'your environment' means memory available,
> operating system, hardware. Usage scenarios? hmmm... please clarify
> with a brief example, thanks. :)

Here's an example: Let's say that your webapp doesn't use HttpSessions
and does no caching. You need to be able to handle 100 simultaneous
connections that do small fetches/inserts from/into a relational
database. Your pages are fairly simple and don't have any kind of
heavyweight app framework taking-up a whole bunch of memory to do
simple things.

For this situation, you can probably get away with a 64MiB heap. If
your webapp uses more than 64MiB, there is probably some kind of
problem. If you only need a 64MiB heap, then you can probably run on
fairly modest hardware: there's no need to lease that 128GiB server
your vendor is trying to talk you into.

On the other hand, maybe you have aggressive caching of data that
benefits from having a large amount of heap space. Or maybe you need
to support 1000 simultaneous connections and need to do XSLT
processing of multi-megabyte XML documents and your XSLTs don't allow
stream-processing of the XML document (oops). Or maybe you have to
keep a large amount of data in users' HttpSession objects (maybe a few
dozen MiB) and you need to support a few thousand simultaneous users
(not connections). 10k users each with a 5MiB heap = 48GiB.

There is no such thing as a "good recommendation" for heap size unless
the person making the recommendation really understands your use case(s).

I generally have these two suggestions that I've found to be
universally reasonable:

1. Make -Xms = -Xmx to eliminate heap thrashing: the JVM is going to
eat-up that large heap space at some point if you have sized things
correctly, so you may as well not make the memory manager have to work
any harder than necessary.

2. Run with the lowest heap space that is reasonable for your
environment. I like doing this because it actually helps you diagnose
things more easily when they go wrong: a small heap yields a smaller
heap-dump file, is GC'd more frequently and therefore contains fewer
long-lived dead objects, and will cause an OOME sooner if you have
some kind of leak. Of course, nobody wants to experience an OOME but
you also don't want to watch a 50GiB heap fill-up 800 bytes at a time
due to a small leak.

I've had people tell me that I should run with the biggest heap I "can
afford" meaning both financially - 'cause you have to buy a bunch of
memory - and reasonably within the constraints of the OS (it's not
reasonably to run a 9.9GiB heap with 10GiB of physical RAM, for
instance). The reasoning is twofold:

1. If you have leaks, they will take a lot more time to blow up.
(Obviously, this is the opposite of my recommendation, but it's worth
mentioning as it's a sound argument. I just disagree with the
conclusion). If you watch the heap-usage profile over time, you can
see it going up and up and instead of getting an OOME, you can predict
when it will happen and bounce the server at your convenience.

2. Since the cost of a GC is related to the number of live objects
during a collection and not the size of the heap (though obviously a
smaller heap can fit fewer live objects!), having a huge heap means
that GCs will occur less frequently and so your total GC throughput
will (at least theoretically) be higher.

A counter-argument to the second #2 above is that short-lived objects
will be collected quickly and long-lived objects will quickly be
promoted to older generations, so after a short period of runtime,
your GCs should get to the point where they are very cheap regardless
of heap size.

> heap settings tailored to 'my' environment and usage... hmmm, not
> many users hitting the app, app is not used 'all day long', app has
> @Schedule tasks that connects to an email acct, downloads  customer
> email requests, and inserts customer requests into database (Martin
> recommended to close resources; sometime ago, I had to refactor all
> of that code, and I am closing the connection to the email acct,
> and open the connection when @Schedule tasks are executed), i am
> using JMS via TomEE/activeMQ to perform some tasks, asynchronously
> (tomee committers told me that use of @Asynchronous would be
> better, and less overhead); honestly, I do open 2 or 3 JMS
> resources/queues in @ApplicationScoped @PostConstruct (if I'm not 
> mistaking) and close those resources in @ApplicationScoped
> @PreDestroy; why? I think I read on ActiveMQ site/documentation,
> where they recommend that that is better on performance, than
> open/close-on-demand.

IMO, batch processes like the one you describe are better done by
specialty schedulers like cron on *NIX and the Task Scheduler on
Windows. That may not be more convenient for you, so I can only tell
you my opinion: webapps should be focused on the web-based portion(s)
of your service offering. Divorcing your batch-style processing from
your webapp will (at least theoretically) make your webapp more stable
and your batch-stuff more flexible (changed your mind about schedule?
No problem, just modify crontab or whatever instead of bouncing your
webapp). You can also easily move that batch-stuff to another server
if it starts to get heavy -- needs lots of CPU, memory, etc. -- and
you don't have to set up a full-block application-server environment
with Tomcat (TomEE), etc.

> Almost forgot...as I mentioned in another thread, as enduser
> changes data, I have an implementation that keeps google calendar
> in sync with the database, which involves JMS/ActiveMQ/MDB and
> many/multiple requests to google calendar API.

Do you do that in a pipelined way (e.g. queuing the updates) or do you
do everything symchronously while the (web) user waits? There are
advantages to both strategies. Obviously this is all off-topic for the
thread. If you're interested in a separate discussion, please open a
new thread.

> hmmm, more about usage, I have the following:
> 
> <Resource id="jdbc/...." type="javax.sql.DataSource"> JdbcDriver
> org.apache.derby.jdbc.EmbeddedDriver JdbcUrl
> jdbc:derby:....;create=true UserName .... Password .... JtaManaged
> true jmxEnabled true InitialSize 2 MaxActive 80 MaxIdle 20 MaxWait
> 10000 minIdle 10 suspectTimeout 60 removeAbandoned true 
> removeAbandonedTimeout 180 timeBetweenEvictionRunsMillis 30000 
> jdbcInterceptors=StatementCache(max=128) </Resource>

That seems like a lot of db resources to me (again: it all depends
upon your user cases). We have a user-load of roughly 20-100 users,
mostly in the US and mostly during "normal" hours (between 06:00 and
23:00 EDT), the webapp is *very* RDBMS-heavy, and we have
maxActive/maxIdle = 20/10. Only under the most extreme circumstances
do we ever have maxWait (10000ms) problems -- usually when we have a
bunch of users all performing the same very-complex query
simultaneously. (We're working on that one ;)

> I do occasionally see the sawtooth-looking graph, and eventually, I
> see the graph even out (non-sawtooth-looking graph).

Well, every request requires objects to be created and ultimately
discarded (e.g. java.lang.String objects to represent your request
parameters, etc.), so you should never see the sawtooth go away
completely. Depending upon the parameters of your graph, you might not
really be able to see it.

> remember, I do restart tomee quite often, especially when I have
> software updates to deploy to/on the server. It's been a while,
> since I let it run a few days without stop-deploy-start. I am
> looking forward to letting it be and running without touching it,
> so I can see if it results in an OOME or reaches the peak.

You could always run a load-test against it in a test bed. JMeter is
your friend.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJRazL5AAoJEBzwKT+lPKRY1AkP/i8fDS/trZEvabWHTR5Ly5TW
zC9T63meZn3KaOwtM7aZSgXeMies8ZCQBjhVm4bwMDIBknt3cDR8WXKshFCGP7eC
LeCHzYJQkfEqfNOkMkb+FGYsXOKyB33HGLlquQ6VdJKq+UQ8Shvr6CjwBmDfgOe2
TC8EyKHcv/AD/3xBQKCfZr9xZvy2Pd+ut37QqLGV4d9I4BfH172B3lhdnav7Ovf5
6y0NCdfmd85QoCFhXNSCCZZOzJ2uiT0yFaEokFcRJuTDPBIpQ1PhLYx/kdTBN09W
tS5maDvQFbr7piIiDsnD2mZwtXKi6n5xQTAB6lBhVSkVNFV8A0Uj5+4n9Aj+sHGP
QHS5jFVn9cY6GljZ5WTbTtOpCiHb8/p1a0kISDnhzg7eabsBA5ONn2hHRVtz0gU3
R/DAgTxh/IGJa5F4AzDPsGEIHZ8gy3kJrGjvmvg/dZCdwrCKSTVArmf188/45NSy
6+KKEuXqTYOiVOy7EqGM8Mg00UX7GLXIKJppLXdMFtJ5YJeqef0/fvWbP5PYxUb8
NgQZmK9rUTKbvPGnV6AAJixDhCj8jv/AkChgunPl20b/PAuioyVSo6X5tCByJd3j
33kSqbdVf89gNb2ZFbqAqVAseCQKB+6+2MMGO23k5M9x0kUmmcb1lmTCKz3e83Yj
JqH0B7XxPzwKKf8lR+KM
=6LvH
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message