Mailing-List: contact tomcat-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <tomcat-user@jakarta.apache.org>
Date: Thu, 26 Feb 2004 12:04:45 -0800 (PST)
From: Oscar Carrillo <tomcat@daydream.stanford.edu>
To: Daniel Gibby <dgibby@edirectpublishing.com>
cc: tomcat-user@jakarta.apache.org
Subject: Re: thread dump analysis
In-Reply-To: <40324D6F.7090307@edirectpublishing.com>
Message-ID: <Pine.LNX.4.44.0402261201040.14712-100000@daydream.stanford.edu>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII

Hi,

Saw your message on the boards.

Did you make sure you have this environment variable set?

On systems that I need it, I put it in my tomcat startup file.

LD_KERNEL_ASSUME="2.2.5"

Check out my howto page for my brief notes on threading and it's potential
problems with JVMs and threads.

http://www.linuxjava.net/howto/webapp/

Oscar

On Tue, 17 Feb 2004, Daniel Gibby wrote:

> My tomcat 4.1.29 instance running J2RE 1.4.1 IBM build 
> cxia321411-20030930 on RedHat 9 kernel 2.4.18-14
> keeps gaining processor usage until finally can't answer requests 
> successfully.
> 
> The machine has a relatively light load.
> 
> I did a kill -3 on the process that showed up on top and got a stack 
> trace... the problem is I have no idea how to analyze the thread dump to 
> see what is consuming CPU.
> I'm sure something must be spinning its wheels, but I don't know how to 
> tell... I can just see that when I run top my tomcat process has 99.9 % 
> of the CPU and the load average is 8.00 8.00 8.00
> 
> I've fixed problems in the past on a separate java application (not 
> tomcat) where I can tell what the problem is in the thread dump because 
> a thread waiting to be notified is also the one that has a lock on it to 
> notify the thing that is waiting to notify it... (that didn't make 
> sense, I know... but anyway it is basically a circle where it won't ever 
> get woken up.)
> However, in this tomcat case, I can't see anything like that where 
> something is waiting in circles... even though I wouldn't rule that out. 
> My experience on reading thread dumps is limited... Anyway, can someone 
> who has better experience tell me what is consuming the CPU? Restarting 
> tomcat brings the load back down, and it slowly goes up again... like 
> over a few days to a weeks time it is back up to 8.00 Load Average.
> 
> I won't include the whole file. I trimmed the file down to 1350 lines by 
> getting rid of a lot of 2HPMEMMAPLINE lines and the section titled:
> 0SECTION       CL subcomponent dump routine
> but I think that is still too long to post here.
> 
> I'm hoping that someone can tell me what to include and what to exclude 
> and I'll reply with the appropriate parts of the dump.
> 
> Thanks,
> Daniel
> 
> NULL           
> ------------------------------------------------------------------------
> 0SECTION       TITLE subcomponent dump routine
> NULL           ===============================
> 1TISIGINFO     signal 3 received
> 1TIDATETIME    Date:                 2004/02/17 at 08:53:22
> 1TIFILENAME    Javacore filename:    /tmp/javacore.20040217.085322.23429.txt
> NULL           
> ------------------------------------------------------------------------
> 0SECTION       XHPI subcomponent dump routine
> NULL           ==============================
> 1HPTIME        Tue Feb 17 08:53:22 2004
> 1HPSIGRECV     SIGQUIT received in ?? at (nil) in ??.
> 1HPFULLVERSION J2RE 1.4.1 IBM build cxia321411-20030930
> NULL
> 1HPOPENV       Operating Environment
> NULL           ---------------------
> 2HPHOSTNAME    Host             : somehost.com.(none)
> 2HPOSLEVEL     OS Level         : 2.4.18-14.#1 Wed Sep 4 13:35:50 EDT 2002
> 2HPLIBCVER     glibc Version    : 2.2.93
> 2HPCPUS        Processors -
> 3HPARCH          Architecture     : (not implemented)
> 3HPNUMCPUS       How Many         : (not implemented)
> 3HPCPUSENABLED   Enabled          : 1
> NULL
> 1HPMEMINFO     Memory Info
> NULL           -----------
> 2HPMEMLINE             total:    used:    free:  shared: buffers:  cached:
> 2HPMEMLINE     Mem:  1055625216 1015181312 40443904        0 83464192 
> 614227968
> 2HPMEMLINE     Swap: 1052827648   929792 1051897856
> 2HPMEMLINE     MemTotal:      1030884 kB
> 2HPMEMLINE     MemFree:         39496 kB
> 2HPMEMLINE     MemShared:           0 kB
> 2HPMEMLINE     Buffers:         81508 kB
> 2HPMEMLINE     Cached:         599596 kB
> 2HPMEMLINE     SwapCached:        236 kB
> 2HPMEMLINE     Active:         552968 kB
> 2HPMEMLINE     Inact_dirty:    344020 kB
> 2HPMEMLINE     Inact_clean:     50304 kB
> 2HPMEMLINE     Inact_target:   189456 kB
> 2HPMEMLINE     HighTotal:      130880 kB
> 2HPMEMLINE     HighFree:         1024 kB
> 2HPMEMLINE     LowTotal:       900004 kB
> 2HPMEMLINE     LowFree:         38472 kB
> 2HPMEMLINE     SwapTotal:     1028152 kB
> 2HPMEMLINE     SwapFree:      1027244 kB
> 2HPMEMLINE     Committed_AS:  1067972 kB
> NULL
> 1HPUSERLIMITS  User Limits (in bytes except for NOFILE and NPROC) -
> NULL           -----------
> 2HPUSERLIMIT   RLIMIT_FSIZE   : infinity
> 2HPUSERLIMIT   RLIMIT_DATA    : infinity
> 2HPUSERLIMIT   RLIMIT_STACK   : 2093056
> 2HPUSERLIMIT   RLIMIT_CORE    : 0
> 2HPUSERLIMIT   RLIMIT_NOFILE  : 1024
> 2HPUSERLIMIT   RLIMIT_NPROC   : 7168
> NULL
> 1HPSIGHANDLERS JVM Signal Handlers
> NULL           -------------------
> 2HPSIGHANDLER  HUP            : unknown handler
> 2HPSIGHANDLER  INT            : unknown handler
> 2HPSIGHANDLER  QUIT           : unknown handler
> 2HPSIGHANDLER  ILL            : unknown handler
> 2HPSIGHANDLER  TRAP           : unknown handler
> 2HPSIGHANDLER  ABRT           : unknown handler
> 2HPSIGHANDLER  FPE            : unknown handler
> 2HPSIGHANDLER  KILL           : default handler
> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org