Return-Path: 
 <users-return-166795-apmail-tomcat-users-archive=tomcat.apache.org@tomcat.apache.org>
Delivered-To: apmail-tomcat-users-archive@www.apache.org
Received: (qmail 22953 invoked from network); 30 Jul 2007 11:08:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 30 Jul 2007 11:08:05 -0000
Received: (qmail 33962 invoked by uid 500); 30 Jul 2007 11:07:51 -0000
Delivered-To: apmail-tomcat-users-archive@tomcat.apache.org
Received: (qmail 33938 invoked by uid 500); 30 Jul 2007 11:07:51 -0000
Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:users-help@tomcat.apache.org>
List-Unsubscribe: <mailto:users-unsubscribe@tomcat.apache.org>
List-Post: <mailto:users@tomcat.apache.org>
List-Id: <users.tomcat.apache.org>
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Delivered-To: mailing list users@tomcat.apache.org
Received: (qmail 33927 invoked by uid 99); 30 Jul 2007 11:07:51 -0000
Received: from Unknown (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Jul 2007 04:07:51 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mark.stevens99@googlemail.com
 designates 209.85.198.185 as permitted sender)
Received: from [209.85.198.185] (HELO rv-out-0910.google.com) (209.85.198.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Jul 2007 11:07:43 +0000
Received: by rv-out-0910.google.com with SMTP id f1so332244rvb
        for <users@tomcat.apache.org>; Mon, 30 Jul 2007 04:07:22 -0700 (PDT)
DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed;
        d=googlemail.com; s=beta;
        h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=NJVfVbWV8UvUOX9kp8U1ndUdzcVEFOIBakLlWDbVyPHgtZa3WIJMIqbyrgSbCob6ufNfBEUwwSB38KJpbMhY2o3OfQOc9EWbRGU1hOHZoQzCDemjnuHEr7XIXqRs+j7upG4wLf65NWks7g3sc17HSDytL8q2FCrVDdx+iG4JVo4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=MbqWt6MAHMxlvKt9e2rWw1CyNzALWbA78+2Dq673VJz3FbrA1CwlEsUG2iB2+WwKzhPuvZ/8d4PySISuXqCIdYSwLmmk/DwXk3EYD3PouKqJ7duFobfokJ3ylFq9FiqSc9zjZBVnW2/FhDJh8L63/vPGtmRXES49qeReqXrZqk8=
Received: by 10.114.175.16 with SMTP id x16mr4447387wae.1185793642711;
        Mon, 30 Jul 2007 04:07:22 -0700 (PDT)
Received: by 10.114.89.3 with HTTP; Mon, 30 Jul 2007 04:07:22 -0700 (PDT)
Message-ID: <6368e94c0707300407t161ad98ehd0c1e0e910c0918c@mail.gmail.com>
Date: Mon, 30 Jul 2007 12:07:22 +0100
From: "Mark Stevens" <mark.stevens99@googlemail.com>
To: "Tomcat Users List" <users@tomcat.apache.org>
Subject: Re: Tomcat consuming entire CPU.
In-Reply-To: <6368e94c0707250916t2cc825v314851520d81e829@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <6368e94c0707240332n377cdbd6hd895ac9afeea9335@mail.gmail.com>
	 <AEF3D568DD1D9E428048C592AA164597061585CE@USRV-EXCH4.na.uis.unisys.com>
	 <6368e94c0707240941x1a6f7fddg12e27469e295937b@mail.gmail.com>
	 <46A6F384.7060909@iki.fi>
	 <6368e94c0707250916t2cc825v314851520d81e829@mail.gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

Hi again,

I think I've located the Thread that is consuming the CPU, although
I'm none the wiser.

Every time I capture the thread using much higher percentage CPU than
all other threads, it's pointing to '

"VM Thread" prio=5 tid=0xb12f8 nid=0x4 runnable

With no stack trace associated to it, is this common?


when trussing the main Tomcat process it's performing hundreds/thousands of

stat64("<jar file including path>", 0xDBE7EDA8) = 0 against the Jar
files in the directory for each jar contained in the webapp I'm
troubleshooting.

I'm generating the load by sending loads of requests for a single
page, it seems like a lot of stat64's for the same thing that I would
assume should be cached in Tomcat unless specified otherwise?

Anyone seen this type of behaviour?


Many thanks.

Mark.






















On 25/07/07, Mark Stevens <mark.stevens99@googlemail.com> wrote:
> Thanks very much all for the advice,
>
> I've managed to get my thread dumps nicely formatted side by side in
> HTML using TDAN (Thread Dump Analyzer)
>
> I'll work out how to then get thread ID's using 'ps' and then should
> be in business.
>
> I'm also going to speak with DBA about limiting Oracles(Database)
> memory, and get Tomcat moved from was into RAM, I think this will help
> general performance, but isn't the root cause.
>
>
>
>
> Thanks again,
>
> Mark.
>
>
>
>
> On 25/07/07, Juha Laiho <Juha.Laiho@iki.fi> wrote:
> > Mark Stevens wrote:
> > > I don't think we are using sendfile, to be honest, I've had no
> > > involvement in development of the code, I'm just requested to look
> > > after the server.
> > >
> > > I'm going to try and work out how to read thread dumps, hopefully this
> > > will help, I'm finding my X11 connection too slow for J-Profiler to be
> > > usable.
> >
> > Ok, some more details on what to look for (pretty much already
> > said by Charles in an earlier message).
> >
> > When you encounter the CPU hogging situation, run (several times, if
> > needed) such form of "ps" command that shows data on the individual
> > threads. This'll give you the thread ID of the CPU-hungry thread (most
> > possibly there is just one thread in this state), when you look at the
> > CPU time used by thread - one is incrementing, others are not.
> >
> > Then run "kill -QUIT <tomcat_process_id>" to get the thread dump in
> > catalina.out logfile.
> >
> > When you have the thread dump, you'll at least get a name of the thread
> > and the Java method name being executed in the thread which is eating
> > your CPU. This gives, if nothing else, a possibility to see whether
> > the thread is looping in Java library code, application code, or Tomcat
> > server code. A caveat on the data formats; at least on Linux platforms
> > the thread ids from "ps" are in decimal, whereas in Java thread dump
> > they are in hexadecimal, so you'll need to do a dec->hex conversion to
> > the thread id from ps before searching in the thread dump.
> >
> > > Something else I noticed in my Live envrionment is that Oracle is
> > > hogging all the memory and tomcat is being forced into swap.
> >
> > Oracle as in database - or application server? If database, then you'll
> > need to get your DBA people to re-tune Oracle to leave some memory for
> > others as well. If you're the DBA as well, see Oracle documentation on
> > tuning SGA and db buffer cache sizes.
> > --
> > ..Juha
> >
> > ---------------------------------------------------------------------
> > To start a new topic, e-mail: users@tomcat.apache.org
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
>

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org