tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Chheda <chetan_chh...@yahoo.com>
Subject Re: Tomcat not closing threads -- SOLVED ??
Date Tue, 26 May 2009 18:47:03 GMT
After some digging thru config files setup by the vendor, I think I might have found the root
cause ..Correlating the access_log and tomcat logs, I found out that tomcat threads were shooting
up whenever a large number of GIF files were being requested. 

These are the modjk settings in our application config files that are included in httpd.conf

JkMount /LCSW/* ajp13   
JkMount /LCSW/*.jsp ajp13
JkMount /LCSW/*.jsp/* ajp13
 
So when an image is accessed via the following URL, http://blahbhal/LCSW/images/header.gif
, the above routes were sending it to tomcat. I verified this by shutting down tomcat and
accessing the above URL, and was unsuccessful.
 
Now I plan to add the following excludes 
JkUnMount /*.htm ajp13
JkUnMount /*.html ajp13
JkUnMount /*.png ajp13
JkUnMount /*.gif ajp13
JkUnMount /*.jpg ajp13

I tested the above in a test environment and was able to access a gif file with tomcat down.


my question is, how does the ajp13 connector interpret a gif file/static content request
as opposed to a jsp request? 

Chetan

 

________________________________
From: Chetan Chheda <chetan_chheda@yahoo.com>
To: Tomcat Users List <users@tomcat.apache.org>
Sent: Monday, May 18, 2009 5:53:07 PM
Subject: Re: Tomcat not closing threads

Hi martin, 

 Thanks for your reply but you have left me confused 

Since Apache is setup as below 
Maxclients = 512 
ThreadsPerChild = 256

Wont it restrict the processes to 2 processes with 256 threads each instead of the ServerLimit(16)
? Looking at the OS , I see 2 httpd processes each with 258 threads 

Thanks,
Chetan


________________________________
From: Martin Gainty <mgainty@hotmail.com>
To: Tomcat Users List <users@tomcat.apache.org>
Sent: Monday, May 18, 2009 4:00:16 PM
Subject: RE: Tomcat not closing threads


good to know
With worker mpm the connection_pool_size defaults to ThreadsPerChild
ThreadsPerChild that in the final gives the MaxClients connections to the Tomcat.
So the default value for connection_pool_size is always ThreadsPerChild

For threaded and hybrid servers (e.g. beos
    or worker) MaxClients restricts
    the total number of threads that will be available to serve clients.
    The default value for beos is 50. 
For
    hybrid MPMs the default value is 16 (ServerLimit) multiplied by the value of
    25 (ThreadsPerChild). Therefore, to increase MaxClients to a value that requires more
than 16 processes,
    you must also raise ServerLimit.
/*******in your case*******/
(MaxClients)MaxClients=16(ServerLimit)*256(ThreadsPerChild)=4096

onto thread calculation:

Description:Minimum number of idle threads available to handle request
spikes
Syntax:MinSpareThreads number
Default:See usage for details
Context:server config
Status:MPM
Module:beos, mpm_netware, mpmt_os2, worker

    Minimum number of idle threads to handle request spikes.
    Different MPMs deal with this directive
    differently.



    worker
    uses a default of MinSpareThreads
    75 and deal with idle threads on a server-wide basis. If
    there aren't enough idle threads in the server then child
    processes are created until the number of idle threads is greater
    than number.


    mpm_netware uses a default of
    MinSpareThreads 10 and, since it is a single-process
    MPM, tracks this on a server-wide bases.


    beos and mpmt_os2 work
    similar to mpm_netware. The default for
    beos is MinSpareThreads 1. For
    mpmt_os2 the default value is 5.
/*********lets assume you havent touched this and MinSpareThreads stays at 75***********/
MaxSpareThreads(this is the parameter that is plaguing your environment)

The range of the MaxSpareThreads value
      is restricted. Apache will correct the given value automatically
      according to the following rules:

      perchild requires MaxSpareThreads to be less or equal than ThreadLimit.mpm_netware
wants the value to be greater than
        MinSpareThreads.For leader, threadpool and
        worker the value must be greater or equal than
        the sum of MinSpareThreads
        and ThreadsPerChild.
MaxSpareThreads >= MinSpareThreads+ThreadsPerChild
MaxSpareThreads >=75+256
MaxSpareThreads must be >= 331

after all that if you still dont have enough SpareThreads i would up (master thread limit)
ThreadLimit to the max
The default value for ThreadLimit is
    1920 when used with mpm_winnt and
    64 when used with the others.



    There is a hard limit of ThreadLimit 20000 (or
      ThreadLimit 15000 with mpm_winnt)
      compiled into the server. This is intended to avoid nasty effects
      caused by typos.

    /****conceivable to start with 1920 and increase by increments of 500 until number of
threads sufficient ***/
BTW: the ceiling on ThreadLimit=15000

In your case 
ThreadLimit=1920

HTH
Martin Gainty 
______________________________________________ 
Jogi és Bizalmassági kinyilatkoztatás/Verzicht und Vertraulichkeitanmerkung/Note de déni
et de confidentialité
Ez az
üzenet bizalmas.  Ha nem ön az akinek szánva volt, akkor kérjük, hogy
jelentse azt nekünk vissza. Semmiféle továbbítása vagy másolatának
készítése nem megengedett.  Ez az üzenet csak ismeret cserét szolgál és
semmiféle jogi alkalmazhatósága sincs.  Mivel az electronikus üzenetek
könnyen megváltoztathatóak, ezért minket semmi felelöség nem terhelhet
ezen üzenet tartalma miatt.

Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten
wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist
unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet
keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen
wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire
prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe
quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information
seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les
email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune
responsabilité pour le contenu fourni.




> Date: Mon, 18 May 2009 12:03:35 -0700
> From: chetan_chheda@yahoo.com
> Subject: Re: Tomcat not closing threads
> To: users@tomcat.apache.org
> 
> Apache is using worker mpm 
> Server version: Apache/2.0.59  HP-UX_Apache-based_Web_Server
> Server built:  Aug 21 2007 13:59:06
> Server's Module Magic Number: 20020903:12
> Server loaded:  APR 0.9.12, APR-UTIL 0.9.12
> Compiled using: APR 0.9.12, APR-UTIL 0.9.12
> Architecture:  64-bit
> Server compiled with....
>  -D APACHE_MPM_DIR="server/mpm/worker"
> 
> 
> 
> 
> ________________________________
> From: Martin Gainty <mgainty@hotmail.com>
> To: Tomcat Users List <users@tomcat.apache.org>
> Sent: Monday, May 18, 2009 2:44:45 PM
> Subject: RE: Tomcat not closing threads
> 
> 
> need to know if Apache MPM prefork or Apache worker is configured
> $APACHE_HOME/bin/Apache -l
> 
> *any compiled-in modules with 'mpm' in the module-name is prefork*
> *any compiled-in modules with 'worker' in the module-name is worker*
> 
> Martin Gainty 
> ______________________________________________ 
> Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
> 
> Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so
bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer
Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und
entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails
koennen wir keine Haftung fuer den Inhalt uebernehmen.
> Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire
prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe
quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information
seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les
email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune
responsabilité pour le contenu fourni.
> 
> 
> 
> 
> > Date: Mon, 18 May 2009 08:45:43 -0700
> > From: chetan_chheda@yahoo.com
> > Subject: Re: Tomcat not closing threads
> > To: users@tomcat.apache.org
> > 
> > So what you are suggesting is my mod_jk and tomcat parameters in server.xml are
messed up .. 
> > 
> > Here is my current setup I have one apache load balancing to 2 tomcat workers
> > Maxclients = 512 
> > ThreadsPerChild = 256
> > Connection_pool_size = 256 
> > Tomcat MaxThreads = 150 
> > 
> > I will suggest my team to change this to ... 
> > connection_pool_size = 120  - which means that I will have 120*2(2 apache server
procs) = 240 threads per tomcat 
> > Tomcat maxthreads = 300
> > 
> > So out of total 512 worker threads , a max of 480 threads will be used by mod_jk
leaving the rest for serving static content. 
> > 
> > is my math correct? 
> > 
> > Thanks!
> > 
> > 
> > 
> > ________________________________
> > From: Martin Gainty <mgainty@hotmail.com>
> > To: Tomcat Users List <users@tomcat.apache.org>
> > Sent: Monday, May 18, 2009 11:23:58 AM
> > Subject: RE: Tomcat not closing threads
> > 
> > 
> > some documentation:
> > 
> > http://httpd.apache.org/docs/2.0/mod/mpm_common.html
> > MaxClients:
> > The MaxClients directive sets the limit on the number of simultaneous requests that
will be served. 
> > Any connection attempts over the MaxClients limit will normally be queued, up to
a number based on the ListenBacklog directive. 
> > Once a child process is freed at the end of a different request, the connection
will then be serviced.
> > For non-threaded servers (i.e., prefork), MaxClients translates into the maximum
number of child processes that will be launched to serve requests. The default value is 256;
to increase it, you must also raise ServerLimit.
> > For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts the total
number of threads that will be available to serve clients. 
> > The default value for beos is 50. 
> > For hybrid MPMs (your situation) the default value is 16 (ServerLimit) multiplied
by the value of ThreadsPerChild. 
> > Therefore, to increase MaxClients to a value that requires more than 16 processes,
you must also raise ServerLimit.
> > 
> > ThreadsPerChild
> > This directive sets the number of threads created by each child process.
> > The child creates these threads at startup and never creates more. 
> > If using an MPM like mpm_winnt, where there is only one child process, this number
should be high enough to handle the entire load of the server. 
> > If using an MPM like worker, where there are multiple child processes, the total
number of threads should be high enough to handle the common load on the server.
> > The default value for ThreadsPerChild is 64 when used with mpm_winnt 
> > The default value is 25 when used with other configurations such as worker
> > 
> > also from mladens discussion of connection_pool_size calcs based on Apache HTTPD
Prefork vs Worker 
> > http://www.mail-archive.com/users@tomcat.apache.org/msg14767.html
> > Apache Prefork configuration
> > Apache 1.3 and Apache 2.x-prefork create a separate child process for
> > each client connection, meaning that if you have default
> > 250 MaxClients, you can end up with MaxClients * connection_pool_size
> > connections to the Tomcat, that would in your case be 250000.
> > (and in the case of ThreadsPerChild parameter MaxClients*64*connection_pool_size)
> > 
> > Apache Worker configuration
> > With worker mpm the connection_pool_size defaults to ThreadsPerChild,
> > that in the final gives the MaxClients connections to the Tomcat.
> > So the default value for connection_pool_size is always ThreadsPerChild
> > that with prefork mpm is by design always 1.
> > (in the case of ThreadsPerChild parameter not being null MaxClients*ThreadsPerChild)
> > 
> > HTH
> > Martin 
> > ______________________________________________ 
> > Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
> > 
> > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein,
so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer
Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und
entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails
koennen wir keine Haftung fuer den Inhalt uebernehmen.
> > Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire
prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe
quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information
seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les
email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune
responsabilité pour le contenu fourni.
> > 
> > 
> > 
> > 
> > Date: Mon, 18 May 2009 07:43:04 -0700
> > From: chetan_chheda@yahoo.com
> > Subject: Re: Tomcat not closing threads
> > To: users@tomcat.apache.org
> > 
> > 
> > 
> > Our network team is already engaged and looking into this. We had several instances
of tomcats locking up today. I have attached one such thread dump here ..
> > 
> > Can some of you pls take a look and point me into the right direction?
> >    
> > 
> > 
> > 
> > 
> > 
> > From: rad muthu <rmuthu22@gmail.com>
> > To: Tomcat Users List <users@tomcat.apache.org>
> > Sent: Wednesday, May 13, 2009 8:05:08 PM
> > Subject: Re: Tomcat not closing threads
> > 
> > Chetan,
> > 
> > 
> > We had similar issue with Sun solaris and weblogic.
> > 
> > My system admin found out there is a socket read errors are going on . He
> > used snoop command in solaris. After tuning the TCP parameters the issue got
> > resolved.
> > 
> > You might also need to something like that.
> > 
> > Thanks
> > Krish
> > 
> > 
> > 
> > On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda <chetan_chheda@yahoo.com>wrote:
> > 
> > > I guess I need to ask my question again. Im primarily an
> > HP-UX
> > > administrator and recently inherited this web based application ...
> > >
> > > We are having some issues in accomodating additional user
> > > load/functionality. While a part of the team is looking at optimizing code,
> > > I am responsible for infrastructure componants.
> > >
> > > One particular focus area for me is the mod_jk and tomcat configurations.
> > > 1. Last of April saw one of the tomcat's stop processing requests. Looking
> > > at the mod_jk.log I saw the following errors
> > > Unable to get the free endpoint for worker XXX  from 37 slots ....
> > > I have setup the status servlet for mod_jk in which I saw that max slots
> > > were opened. Thinking that mod_jk needs more number of connections, I
> > > basically set it to ThreadsPerChild from worker MPM
> > >
> > > 2. Now we are randomly seeing that even a simple activity as user login
> > > causes a spike in used tomcat connector threads.
> > Eventually all tomcat
> > > threads are used and no new requests can be serviced.
> > >
> > > So, long story short, is my apache, mod_jk and tomcat configuration in
> > > sync?
> > > Maxclients = 512
> > > ThreadsPerChild = 256
> > > So that means 2 server processes with 256 threads each.
> > >
> > > I have 2 worker threads in my worker.properties file with
> > > connection_pool_size=256.
> > > Does this mean that total number of connections into tomcat = 256 *
> > > 2(number of workers) * 2(number of server procs) = 1024 ??
> > >
> > > Do I need to change the connection_pool_size to 120 . Which means 120*2 *2
> > > = 480 and that leaves the remainder of apache threads for static content?
> > >
> > > 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in
> > > server.xml . Is this why connections remain open?
> > >
> > > Thanks,
> > > Chetan
> > >
> > >
> > >
> > ________________________________
> > > From: "Caldarale, Charles R" <Chuck.Caldarale@unisys.com>
> > > To: Tomcat Users List <users@tomcat.apache.org>
> > > Sent: Tuesday, May 12, 2009 10:47:00 PM
> > > Subject: RE: Tomcat not closing threads
> > >
> > > > From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> > > > Subject: Re: Tomcat not closing threads
> > > >
> > > > I managed to get a thread dump during one such tomcat hangs. Most of
> > > > the threads are in the following status ...
> > >
> > > The ones you show are simply waiting for input from httpd; that's a pretty
> > > normal state.  When looking at a thread dump, it's often
> > the oddball thread
> > > that's not doing what the rest are that is causing a problem.  However, if
> > > all of the threads are just waiting to receive something from httpd, then
> > > you may have to look outside of Tomcat for the cause.
> > >
> > > - Chuck
> > >
> > >
> > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> > > MATERIAL and is thus for use only by the intended recipient. If you received
> > > this in error, please contact the sender and delete the e-mail and its
> > > attachments from all computers.
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > > For additional commands, e-mail: users-help@tomcat.apache.org
> > >
> > >
> > >
> > >
> > 
> > 
> > _________________________________________________________________
> > Windows Live™: Keep your life in sync.
> > http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009
> > 
> > 
> >      
> 
> _________________________________________________________________
> Hotmail® goes with you. 
> http://windowslive.com/Tutorial/Hotmail/Mobile?ocid=TXT_TAGLM_WL_HM_Tutorial_Mobile1_052009
> 
> 
>      

_________________________________________________________________
Hotmail® has ever-growing storage! Don’t worry about storage limits.
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutorial_Storage1_052009


      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message