Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (nike.apache.org: local policy)
Date: Thu, 7 Jun 2012 14:02:34 -0700 (PDT)
From: Daniel Mikusa <dmikusa@vmware.com>
To: Tomcat Users List <users@tomcat.apache.org>
Message-ID: <2992259.465.1339102948846.JavaMail.daniel@cleveland>
In-Reply-To: 
 <CANj+m=g4F8=hX30XNaT6qUUQfsXH0vh5Lt5dVRWPtigEAV6RRA@mail.gmail.com>
Subject: Re: Java process killed by oom-killer in Ubuntu
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

----- Original Message -----
> Only 52 java threads.  It used to fluctuate more (we made some
> changes
> to the app to perform a task in a single thread rather than spawning
> multiple threads, but the crash still occurs) . The number of threads
> is always below 100.
>=20
>  jstack -F 21370 | grep ^Thread | wc -l
>  ps -T -p 21370   (This gives me 63)
>=20
> I don't seem to specify the -Xss option:

In some applications with a large number of threads (particularly when runn=
ing on 64-bit hardware) this setting can cause a problems.  The default val=
ue is pretty large (I think it's 1M on 64-bit systems).  Since most apps do=
n't need that large of a value, an easy performance tuning step is to lower=
 the value of -Xss.

Since you don't have very many threads, it seems unlikely that this is caus=
ing your problem though.

That being said, you could try explicitly setting a value for the thread st=
ack size.  Finding the right values takes some testing though.  I usually s=
tart with something like 192k and run a few application tests.  If I see an=
y stack overflow exceptions then I increase the value and rerun the tests. =
 Repeat until there are no stack overflow exceptions.


On a different note, what is the specific version of the JVM that you are r=
unning?  If it's not the latest, you could always try upgrading to the late=
st version.

Dan


>=20
> Xms6g -Xmx6g -XX:NewSize=3D4G -XX:MaxNewSize=3D4G -XX:SurvivorRatio=3D6
> -XX:MaxPermSize=3D512M -XX:-UseConcMarkSweepGC -XX:+UseStringCache
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=3D/home/example/logs
>=20
> -Jorge
>=20
>=20
>=20
>=20
>=20
>=20
> On Thu, Jun 7, 2012 at 12:07 PM, Daniel Mikusa <dmikusa@vmware.com>
> wrote:
> > ----- Original Message -----
> >> I am using MongoDB through the Java driver allowing up to 100
> >> connections to the MongoDB server.
> >> I also use DBCP with a max size of 50 JDBC connections.
> >> My webapp uses about 150 JAR files.
> >> There is no native libraries loaded from my webapp as far as I
> >> know.
> >> All the app is pure Java code. =C2=A0(Nevertheless, Tomcat is using th=
e
> >> Tomcat Native Library)
> >>
> >> Is there a way I can monitor the number of file descriptors in use
> >> by
> >> the app?
> >>
> >> I have monitored the number of threads, but I haven't seen
> >> anything
> >> unusual.
> >
> > How many threads have you observed? =C2=A0Total threads, not just
> > threads for the connector.
> >
> > Also, what is the value you are using for thread stack size? =C2=A0-Xss
> >
> > Dan
> >
> >
> >
> > (but it could be that the burst is too fast to get catch by
> >> the monitoring tool)
> >>
> >> -Jorge
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Jun 7, 2012 at 11:44 AM, Christopher Schultz
> >> <chris@christopherschultz.net> wrote:
> >> > -----BEGIN PGP SIGNED MESSAGE-----
> >> > Hash: SHA1
> >> >
> >> > Jorge,
> >> >
> >> > On 6/6/12 5:33 PM, Jorge Medina wrote:
> >> >> The web application uses Spring/Postgres/Mongo.
> >> >
> >> > Are you using MongoDB in-process or anything weird like that? Or
> >> > are
> >> > you connecting through some socket-based (or other) API?
> >> >
> >> >> It looks like a memory leak in native code, not java code; so
> >> >> my
> >> >> usual java toolset is not useful.
> >> >
> >> > If what you are observing is accurate (non-heap memory grows,
> >> > heap
> >> > stays reasonable) then it will definitely be more difficult to
> >> > track-down.
> >> >
> >> >> Tomcat runs behind nginx in a EC2 instance. The application
> >> >> uses
> >> >> Sun (now Oracle) JDK 1.6.
> >> >>
> >> >> Any suggestions on what should I look at?
> >> >
> >> > What do your <Connectors> look like? How many JDBC connections
> >> > do
> >> > you
> >> > have in your connection pool (which you are hopefully using!)?
> >> > How
> >> > about the same equivalent for MongoDB?
> >> >
> >> > Does your webapp keep lots of files open? Do you have an
> >> > unusually-large number of JAR files in your webapp? Do you have
> >> > any
> >> > native libraries in use within your webapp?
> >> >
> >> > What are all the non-default system properties that you are
> >> > setting
> >> > at
> >> > JVM launch time (you can easily see this from a 'ps' list)?
> >> >
> >> > Two things that can eat-up native memory fast in a JVM are file
> >> > descriptors and threads, so let's start there.
> >> >
> >> > - -chris
> >> > -----BEGIN PGP SIGNATURE-----
> >> > Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> >> > Comment: GPGTools - http://gpgtools.org
> >> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> >> >
> >> > iEYEARECAAYFAk/Q9ooACgkQ9CaO5/Lv0PDPyQCfVtddxMDOgQbjmMGC3gvnK+Qq
> >> > aZMAnjVu67+9Sm2bdYzAd91ZOrYo3DFI
> >> > =3Dr+vl
> >> > -----END PGP SIGNATURE-----
> >> >
> >> > --------------------------------------------------------------------=
-
> >> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> > For additional commands, e-mail: users-help@tomcat.apache.org
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>=20
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org