tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Glenn Nielsen <gl...@mail.more.net>
Subject Re: Ajp13Processor starting background threads under low load
Date Sun, 20 Oct 2002 12:01:57 GMT
Once an Apache process opens a socket connection to Tomcat for Ajp that
connection remains open until that httpd process dies.  That way the
httpd process can reuse that connection.  For Apache 1.3 there is a
one to one mapping between apache httpd processes and Tomcat Ajp13Processors.
You can end up with as many Ajp13Processors as you have httpd processes.
So the MaxProcessors config needs to be greater than the max number
of httpd processes used by apache.  And each Ajp13Processor runs in
its own thread waiting for a request from Apache.

Regards,

Glenn



Lindsay Patten wrote:
> Thanks for the pointer Glenn, it was much appreciated!
> 
> So, I've been doing thread stacktrace dumps, but what I found doesn't 
> make a lot of sense to me.
> After restarting tomcat there were 5 processor threads all blocked on a 
> monitor:
> 
> "Ajp13Processor[8085][0]" daemon prio=1 tid=0x0x4f801b28 nid=0x45ae 
> waiting on m
> onitor [bd9ff000..bd9ff8ac]
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
>        at java.lang.Object.wait(Object.java:426)
>        at 
> org.apache.ajp.tomcat4.Ajp13Processor.await(Ajp13Processor.java:305)
>        - locked <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
>        at 
> org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:490)
>        at java.lang.Thread.run(Thread.java:536)
> 
> I presume these are waiting for the scheduler thread to pass them a 
> request.
> 
> A while later a bunch of background threads get spawned and I do another 
> dump/trace,
> now there are a whole bunch of threads in a runnable state trying to 
> read request packets:
> 
> "Ajp13Processor[8085][21]" daemon prio=1 tid=0x0x8289b38 nid=0x70d6 
> runnable [ba
> fff000..bafff8ac]
>        at java.net.SocketInputStream.socketRead0(Native Method)
>        at java.net.SocketInputStream.read(SocketInputStream.java:116)
>        at org.apache.ajp.Ajp13.readN(Ajp13.java:429)
>        at org.apache.ajp.Ajp13.receive(Ajp13.java:469)
>        at org.apache.ajp.Ajp13.receiveNextRequest(Ajp13.java:274)
>        at 
> org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:371
> )
>        at 
> org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:495)
>        at java.lang.Thread.run(Thread.java:536)
> 
> Despite the name receiveNextRequest, it looks like that method is being 
> called to receive each request including the first. So, the ajp13 
> processor is sitting blocked waiting for a full ajp13 packet from apache 
> and apache is not sending data and not closing the connection/socket. 
>  From the comments in the code it sounds like apache restarts 
> periodically, closing the sockets and unblocking the read calls, at 
> which point these processors return to the pool - so if apache restarts 
> before you reach your threads limit you are ok.
> 
> What I don't understand is how apache can be starting up ajp13 
> connections to tomcat and then not sending a full packet without this 
> causing user visible failures. Shouldn't there be users getting page 
> loads timing out on them?  That doesn't seem to be happening.
> 
> Does this indicate a fault in apache or is it normal for apache to send 
> partial requests without closing the connection?
> 
> Any further guidance?
> 
> Thanks,
>    Lindsay
> 
> 
> Glenn Nielsen wrote:
> 
>> One way to start debugging this type of problem is to tell the java
>> process running Tomcat to do a Thread Stacktrace Dump.
>>
>> kill -QUIT <java processid>
>>
>> Then analyze the stack traces for all threads.
>>
>> Regards,
>>
>> Glenn
>>
>>
>> Lindsay Patten wrote:
>>
>>> Hi,
>>> I thought I would take a different tack on my problem with the 
>>> Ajp13Processor using up all of its threads problem.
>>> Does anyone know if the Ajp13Processor has a timeout when it is 
>>> looking for a worker to handle a request? If the worker threads were 
>>> all swapped out and taking a long time to get going and respond, 
>>> would the main thread start a new background thread instead of 
>>> waiting? Or does it just have a list of available workers and blocks 
>>> until the worker it selects responds?
>>> If it is the former that would explain my problem since I have 
>>> experienced my processes getting swapped out and taking a long time 
>>> to respond. The question would then become: is a way to specify how 
>>> long the scheduler should wait before starting a new thread? At least 
>>> I would have something I could take to my service provider. On the 
>>> other hand if I knew for sure that the scheduler thread didn't have a 
>>> timeout on worker threads than I could concentrate on looking 
>>> elsewhere.  Does anyone know? Or know where to look or ask short of 
>>> reading the source?
>>>
>>> Thanks,
>>>    Lindsay
>>>
>>> Lindsay Patten wrote:
>>>
>>>> Hi,
>>>> I am having a problem where tomcat keeps intermitantly starting new 
>>>> ajp13 processor threads, eventually it reaches the max and starts 
>>>> refusing connections. My hosting provider (linux machine) provides a 
>>>> private version of tomcat for each user with a single apache web 
>>>> server, there are typically a large number of processes on the 
>>>> machine (>4000). My particular tomcat server (v4.0.3) is not being 
>>>> loaded at all but the ajp13processor threads don't seem to not get 
>>>> reused under some circumstances that I don't understand. I have 
>>>> added logging statements in my jsp pages and the pages appear to run 
>>>> to completion. Each jsp page accesses some info from a mysql db and 
>>>> displays it (*Summary) or takes some data from a form and sticks it 
>>>> in the db (Page). The system will often run for several hours and 
>>>> hundreds of requests without starting any threads, but then 
>>>> sometimes, as below, it starts new threads for almost every request 
>>>> for a while. Is there a way to determine if the threads are getting 
>>>> hung up or returned to the pool? If a thread were swapped out and 
>>>> taking a long time to get swapped back in would tomcat start a new 
>>>> thread? The ps command indicates that the processes are there and 
>>>> sleeping. Any help or debugging pointers would be greatly 
>>>> appreciated.  Thanks - Lindsay
>>>>
>>>>    2002-10-18 03:50:45 Ajp13Processor[8085][14] Starting background 
>>>> thread
>>>>    2002-10-18 03:50:53 Ajp13Processor[8085][15] Starting background 
>>>> thread
>>>>    2002-10-18 03:50:59 NationalSummary start
>>>>    2002-10-18 03:51:00 NationalSummary end
>>>>    2002-10-18 03:51:10 DistrictSummary start
>>>>    2002-10-18 03:51:10 DistrictSummary end
>>>>    2002-10-18 03:51:14 Ajp13Processor[8085][16] Starting background 
>>>> thread
>>>>    2002-10-18 03:51:14 SubdistrictSummary start 42
>>>>    2002-10-18 03:51:14 SubdistrictSummary end 42
>>>>    2002-10-18 03:51:23 Ajp13Processor[8085][17] Starting background 
>>>> thread
>>>>    2002-10-18 03:51:23 PageFrame start
>>>>    2002-10-18 03:51:23 PageFrame end
>>>>    2002-10-18 03:51:24 Ajp13Processor[8085][18] Starting background 
>>>> thread
>>>>    2002-10-18 03:51:24 Page start 206
>>>>    2002-10-18 03:51:24 Page end 206
>>>>    2002-10-18 03:53:32 Ajp13Processor[8085][19] Starting background 
>>>> thread
>>>>    2002-10-18 03:53:32 Page start 206
>>>>    2002-10-18 03:53:33 Page end 206
>>>>    2002-10-18 03:53:56 Ajp13Processor[8085][20] Starting background 
>>>> thread
>>>>    2002-10-18 03:53:56 NationalSummary start
>>>>    2002-10-18 03:53:56 NationalSummary end
>>>>    2002-10-18 03:54:42 Page start 206
>>>>    2002-10-18 03:54:42 Page end 206
>>>>    2002-10-18 03:55:06 DistrictSummary start
>>>>    2002-10-18 03:55:06 DistrictSummary end
>>>>    2002-10-18 03:55:24 Ajp13Processor[8085][21] Starting background 
>>>> thread
>>>>    2002-10-18 03:55:24 NationalSummary start
>>>>    2002-10-18 03:55:24 NationalSummary end
>>>>    2002-10-18 03:55:34 DistrictSummary start
>>>>    2002-10-18 03:55:34 DistrictSummary end
>>>>    2002-10-18 03:55:41 Page start 206
>>>>    2002-10-18 03:55:41 Page end 206
>>>>    2002-10-18 03:56:52 Ajp13Processor[8085][22] Starting background 
>>>> thread
>>>>    2002-10-18 03:56:52 Page start 206
>>>>    2002-10-18 03:56:52 Page end 206
>>>>    2002-10-18 03:58:16 Page start 206
>>>>    2002-10-18 03:58:16 Page end 206
>>>>    2002-10-18 03:59:31 Page start 206
>>>>    2002-10-18 03:59:31 Page end 206
>>>>    2002-10-18 04:00:24 Page start 206
>>>>    2002-10-18 04:00:24 Page end 206
>>>>    2002-10-18 04:01:34 Page start 206
>>>>    2002-10-18 04:01:34 Page end 206
>>>>


--
To unsubscribe, e-mail:   <mailto:tomcat-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:tomcat-user-help@jakarta.apache.org>


Mime
View raw message