tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Mikusa <dmik...@vmware.com>
Subject Re: Java process killed by oom-killer in Ubuntu
Date Thu, 07 Jun 2012 21:02:34 GMT
----- Original Message -----
> Only 52 java threads.  It used to fluctuate more (we made some
> changes
> to the app to perform a task in a single thread rather than spawning
> multiple threads, but the crash still occurs) . The number of threads
> is always below 100.
> 
>  jstack -F 21370 | grep ^Thread | wc -l
>  ps -T -p 21370   (This gives me 63)
> 
> I don't seem to specify the -Xss option:

In some applications with a large number of threads (particularly when running on 64-bit hardware)
this setting can cause a problems.  The default value is pretty large (I think it's 1M on
64-bit systems).  Since most apps don't need that large of a value, an easy performance tuning
step is to lower the value of -Xss.

Since you don't have very many threads, it seems unlikely that this is causing your problem
though.

That being said, you could try explicitly setting a value for the thread stack size.  Finding
the right values takes some testing though.  I usually start with something like 192k and
run a few application tests.  If I see any stack overflow exceptions then I increase the value
and rerun the tests.  Repeat until there are no stack overflow exceptions.


On a different note, what is the specific version of the JVM that you are running?  If it's
not the latest, you could always try upgrading to the latest version.

Dan


> 
> Xms6g -Xmx6g -XX:NewSize=4G -XX:MaxNewSize=4G -XX:SurvivorRatio=6
> -XX:MaxPermSize=512M -XX:-UseConcMarkSweepGC -XX:+UseStringCache
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/example/logs
> 
> -Jorge
> 
> 
> 
> 
> 
> 
> On Thu, Jun 7, 2012 at 12:07 PM, Daniel Mikusa <dmikusa@vmware.com>
> wrote:
> > ----- Original Message -----
> >> I am using MongoDB through the Java driver allowing up to 100
> >> connections to the MongoDB server.
> >> I also use DBCP with a max size of 50 JDBC connections.
> >> My webapp uses about 150 JAR files.
> >> There is no native libraries loaded from my webapp as far as I
> >> know.
> >> All the app is pure Java code.  (Nevertheless, Tomcat is using the
> >> Tomcat Native Library)
> >>
> >> Is there a way I can monitor the number of file descriptors in use
> >> by
> >> the app?
> >>
> >> I have monitored the number of threads, but I haven't seen
> >> anything
> >> unusual.
> >
> > How many threads have you observed?  Total threads, not just
> > threads for the connector.
> >
> > Also, what is the value you are using for thread stack size?  -Xss
> >
> > Dan
> >
> >
> >
> > (but it could be that the burst is too fast to get catch by
> >> the monitoring tool)
> >>
> >> -Jorge
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Jun 7, 2012 at 11:44 AM, Christopher Schultz
> >> <chris@christopherschultz.net> wrote:
> >> > -----BEGIN PGP SIGNED MESSAGE-----
> >> > Hash: SHA1
> >> >
> >> > Jorge,
> >> >
> >> > On 6/6/12 5:33 PM, Jorge Medina wrote:
> >> >> The web application uses Spring/Postgres/Mongo.
> >> >
> >> > Are you using MongoDB in-process or anything weird like that? Or
> >> > are
> >> > you connecting through some socket-based (or other) API?
> >> >
> >> >> It looks like a memory leak in native code, not java code; so
> >> >> my
> >> >> usual java toolset is not useful.
> >> >
> >> > If what you are observing is accurate (non-heap memory grows,
> >> > heap
> >> > stays reasonable) then it will definitely be more difficult to
> >> > track-down.
> >> >
> >> >> Tomcat runs behind nginx in a EC2 instance. The application
> >> >> uses
> >> >> Sun (now Oracle) JDK 1.6.
> >> >>
> >> >> Any suggestions on what should I look at?
> >> >
> >> > What do your <Connectors> look like? How many JDBC connections
> >> > do
> >> > you
> >> > have in your connection pool (which you are hopefully using!)?
> >> > How
> >> > about the same equivalent for MongoDB?
> >> >
> >> > Does your webapp keep lots of files open? Do you have an
> >> > unusually-large number of JAR files in your webapp? Do you have
> >> > any
> >> > native libraries in use within your webapp?
> >> >
> >> > What are all the non-default system properties that you are
> >> > setting
> >> > at
> >> > JVM launch time (you can easily see this from a 'ps' list)?
> >> >
> >> > Two things that can eat-up native memory fast in a JVM are file
> >> > descriptors and threads, so let's start there.
> >> >
> >> > - -chris
> >> > -----BEGIN PGP SIGNATURE-----
> >> > Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> >> > Comment: GPGTools - http://gpgtools.org
> >> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> >> >
> >> > iEYEARECAAYFAk/Q9ooACgkQ9CaO5/Lv0PDPyQCfVtddxMDOgQbjmMGC3gvnK+Qq
> >> > aZMAnjVu67+9Sm2bdYzAd91ZOrYo3DFI
> >> > =r+vl
> >> > -----END PGP SIGNATURE-----
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> > For additional commands, e-mail: users-help@tomcat.apache.org
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message