tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Badi <ki...@poonam.org>
Subject Re: Tomcat - OOM Perm gen
Date Sun, 05 Jul 2015 14:24:44 GMT
Sorry for taking time to reply Chris.

Are you periodically hot-re-deploying your application in production?
If so, you may want to stop doing that, as it appears that you have a
ClassLoader-pinning leak in your application or some dependent library

I do not do hot deployment in production.But somehow I feel its leaking
memory.There are some threads which for some reasons remains hanging.
But I think I will focus on it later on. I do have copy of your
presentation.I will go through it and may be post couple of question if I
have.

You should be able to run a reasonable service in half a gig of heap
space. Just be aware that caching information in your application is
going to significantly increase the amount of heap space required by
your application in a steady-state.

Yup I am aware of this limitation. I know cache is often stored in local
heap and probably that's reason it consumes heap space.
Currently I am checking if ehcache or jcs meets my need. Ehcache for some
reasons fills my heap very fast.I tried storing some 100 json strings and
retrieving them.
My OOM's started once I integrated this library in my web inf /lib
folder.After I removed it, frequency of OOM has substantially
decreased.Probably I am not using it correctly somewhere.

Do you know what open source caching people frequently use for java web
apps ?

Fetching a lot of data isn't usually a big deal. Just make sure that
your "fetch size" is set to something reasonable. There are some JDBC
drivers (MySQL Connector/J in particular) that will load the entire
ResultSet into memory before allowing you to traverse it unless you
make arrangements to limit that memory usage.

But bringing-back thousands of records from a db isn't a problem --
unless you don't *have to* in which case you might want to optimize
your queries to avoid pulling data you don't actually need.

But too-many-records would be a performance problem, not a memory proble
m.

haha.Its fetch size which my autocomplete was missing...I am not sure how I
missed it but it was missing.
Thanks Mark.



On Tue, Jun 30, 2015 at 1:17 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Kiran,
>
> On 6/29/15 10:21 AM, Kiran Badi wrote:
> > Christopher Schultz wrote:
> >> The number of users shouldn't impact your PermGen space. Perhaps
> >> only once you get to that stage are you hitting enough of your
> >> features to load classes into PermGen. (Or maybe you are using
> >> String.intern a lot...)
> >
> > I analysed some logs and I could see that users query features
> > which makes DB calls, so those calls do have 1000's of rows in
> > it.But some calls also fetch empty result set and some error out,
> > partly because code for those calls are broken( Some of those dao
> > classes have hard coded DB parameter which I am cleaning it out
> > now). As far as I know I do not do any string cancat, those calls
> > are all simple list fetch calls to views.
> >
> > I am trying to implement some caching using either ehcache or JCS
> > but I think it has to wait for some time, till I gain some
> > understanding on how these works.( I think I need to serialize lot
> > of model classes for that probably will require some code changes
> > again).i  know I have lot of work to do ,maybe I one at a time
> > change :)
> >
> >> PermGen failures will effect the whole JVM. There is no way to
> >> protect App B from App A unless they are in different JVMs.
> >
> > I can understand this. so doing daily restart now to manage  issue
> > till I figure out some solution to it.
> >
> >> What makes you say that? It seems that you have more information
> >> than you are giving us.
> >
> > Its not hardened code so I think it still has some issues with it.
> > Also during development I can see similar errors on local dev box,
> > If I do deploy and redeploy at least 8 to 10 times, I start seeing
> > those  perm gen errors,its just that it references a new class
> > file every time,maybe I can share it with you all once I get it
> > again.
>
> Stop right there.
>
> Are you periodically hot-re-deploying your application in production?
> If so, you may want to stop doing that, as it appears that you have a
> ClassLoader-pinning leak in your application or some dependent library.
>
> See this presentation for more information:
> http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60
> mins.pdf
>
> > Also it's I have written this code and I am not that fantastic
> > coder yet, but I will reach there short span :)
> >
> >
> >> Usually, PermGen doesn't have to be enormous. What's your memory
> >> cap with your hosting provider?
> >
> > I have private tomcat 7x but I remember hosting provider
> > mentioning that 512mb is final,but I will check with them again
> > later this week.
>
> You should be able to run a reasonable service in half a gig of heap
> space. Just be aware that caching information in your application is
> going to significantly increase the amount of heap space required by
> your application in a steady-state.
>
> > Below is what I see in catalina logs when I do restart of tomcat,
> >
> > Picked up _JAVA_OPTIONS: -Xms20m  -Xmx128m -XX:MinHeapFreeRatio=20
> >  -XX:MaxHeapFreeRatio=40 -XX:NewSize=10m  -XX:MaxNewSize=10m
> > -XX:SurvivorRatio=6 -XX:TargetSurvivorRatio=80
> > -XX:+CMSClassUnloadingEnabled -XX:+CMSClassUnloadingEnabled
>
> So you have a 128MiB max heap space, but you aren't specifying a
> PermGen size. For some frameworks (particularly Spring), you'll need
> to increase the size of the PermGen heap space because it really just
> does need to load 3e8 classes or whatever.
>
> Or, you could upgrade to Java 8 which does not have a PermGen space.
> You'll just eventually run out of "regular" heap space in that case,
> if you really do have a memory leak.
>
> > I think was thinking CMSClassUnloadingEnabled should fix my perm
> > gen issues, but I think its not the case.
>
> Class unloading should pretty much always be enabled, so that setting
> likely does nothing for you. It doesn't look like you are enabling the
> CMS GC, so those settings probably don't make a bit of difference.
>
> >> You either need more PermGen space, or you need to locate some
> >> kind of leak in your application and fix it. IIRC, there are
> >> some RMI-related leaks and Proxy-related leaks in PermGen
> >> depending upon your exact circumstances.
> >>
> >> It would be good to know what's in PermGen when it hits its
> >> limit.
> >>
> >> What are your current heap settings, including PermGen? What
> >> JVM?
> >>
> >> Try: $ jmap -heap <pid>
> >>
> >> and
> >>
> >> $ jmap -permstat <pid>
> >
> > I will try to get those dumps but I do not use any RMI or generate
> > some kind of proxies . Mine is simple app with lot of forms in it.
> > Though there are few calls which fetches lot of data from
> > servers.Sometimes few autocomplete calls fetch 1000's of records.I
> > am trying to remove those calls.
>
> Fetching a lot of data isn't usually a big deal. Just make sure that
> your "fetch size" is set to something reasonable. There are some JDBC
> drivers (MySQL Connector/J in particular) that will load the entire
> ResultSet into memory before allowing you to traverse it unless you
> make arrangements to limit that memory usage.
>
> But bringing-back thousands of records from a db isn't a problem --
> unless you don't *have to* in which case you might want to optimize
> your queries to avoid pulling data you don't actually need.
>
> But too-many-records would be a performance problem, not a memory proble
> m.
>
> > Below is my jvm details
> >
> > Apache Tomcat/7.0.50 1.7.0_17-b02 Oracle Corporation Linux
> > 2.6.32-531.29.2.lve1.3.11.1.el6.x86_64 amd64
> >
> > Thanks Chris for reply.
>
> Hope that helps,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJVks88AAoJEBzwKT+lPKRYOh8P/2DwjRn0hH/R3Xsa0Sf+FFKh
> 5/dzcnRf7C1PvCVQ8IrBTIsKuMCBY6dbS9PxJe7+hWo/UIaScF62B6S4xwd4WCsa
> 5ojVDKslt/BcNGk/6bXyK2uDFXJRFB3j3654XgsU9/T+GvNOF9F6rNQYnRDZVuuv
> UMbphYOEHfFBplIEJnu5Q6SiLD219aKLa3Xmbl1+WHn1tQyoPivFmdBBkAkUO+92
> WN6vwRKiI0VgKZ9mbH2pgI1BVq3qDxKksPx8nv1ju3nDb5MRCWgbNGeJFhl87B5b
> 78XmiNdmYQA4K1PSk4tGXergTDpgJ4PtYhBSYvZC8xqFQnEaiUWtqFwKCqAyFL+S
> FX1Tef/Rc/20XYnxqUm9Wi5XX2Nq2KCBxs/ob7DdwC52TyB9LQDNP7eUvYPZQv13
> cuEW+eDozfYaao9Io+rparQ3Wz95y/1bwgEP5BjxK0b5rqDuDn84HsC6FrSLBPJc
> qp1q6vs4JYB3jIwdNKN+JbGoPTPYfwcI2ESuviVbAAKO+nYQKj0lIgERQBp2UDCN
> RSVSlXW64W//LFuDZzfN+v099vj0BOh1EEWoM+lXDeT9PSpam+gEcaze/gTK5L06
> 0x/0gXgiQGNNgDFmP6ovTpcBA59IA6o8cfYTEbovnJnBr/Bx10pET9D8/QR515od
> XJyzmMIXwXltaOzdAiEe
> =p9EI
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message