Return-Path: Delivered-To: apmail-httpd-dev-archive@httpd.apache.org Received: (qmail 17498 invoked by uid 500); 29 Apr 2002 20:37:17 -0000 Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: dev@httpd.apache.org list-help: list-unsubscribe: list-post: Delivered-To: mailing list dev@httpd.apache.org Received: (qmail 17485 invoked from network); 29 Apr 2002 20:37:16 -0000 Message-ID: <3CCDAEFB.6050705@remulak.net> Date: Mon, 29 Apr 2002 16:37:15 -0400 From: "Paul J. Reder" User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.2.1) Gecko/20010901 X-Accept-Language: en-us MIME-Version: 1.0 To: dev@httpd.apache.org Subject: Re: [PATCH] Possible fix for worker MPM performance problem (Updated patch) References: <004c01c1ec6e$338f94d0$50381b09@sashimi> <007c01c1ed33$1b079000$50381b09@sashimi> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N I have just finished testing cvs-head against cvs-head + Bill's patch to delay listener creation. I believe that this patch is still *very* useful and should be applied. Here is what I did: Started an Apache with the following config: StartServers 1 ThreadsPerChild 64 MaxClients 1024 MinSpareThreads 64 MaxSpareThreads 192 MaxRequestsPerChild 500000 Then I started 26 copies of Jeff Trawick's testing program "b" in 7 different machines (all on a private local network). I used the following parameters for the calls to "b": ./b -c 15 -n 120000 -f replay_file_XXX -v where XXX is a file from 000 to 026, each of which contains a 120000 line segment from an actual apache.org 24 hour access log. Summary of results: cvs-head displayed an immense amount of process/thread churn (repeatedly starting more than it needed to, then killing more than it needed to...). This churn continued during the entire 45-60 minutes of each of the 3 runs I did. cvs-head + Bill's patch displayed very stable process/thread characteristics right from the start. Growing to handle incoming requests and killing an appropriate number of processes/threads to keep within bounds. Overall, cvs-head + Bill's had higher total traffic with higher requests/per second and lower number of requests currently being processed. cvs-head + Bill's did display a higher cpu load, but I believe this relates to the fact that cvs-head + Bill's is spending a lot less time doing task switching and other overhead related to having more (or less) processes/threads than are needed. Now for the long version, with data for your purusing pleasure... ================================================================= cvs-head =================================== Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 13:48:50 Current Time: Monday, 29-Apr-2002 16:34:12 EDT Restart Time: Monday, 29-Apr-2002 16:33:28 EDT Parent Server Generation: 0 Server uptime: 44 seconds Total accesses: 4935 - Total Traffic: 52.7 MB CPU Usage: u3.28 s11.69 cu0 cs0 - 34% CPU load 112 requests/sec - 1.2 MB/second - 10.9 kB/request 142 requests currently being processed, 49 idle workers _WWRRR_WWRW_RWW_____R_RWWWW_WWWW_W_WWWW_WWWW___RW__W_W_R__WRW__R WWWW__WWWWWWW_W_WRRWW_W__WW__WW_W_W_WW_WW_WW__WW_RWWRRWW_W____W_ WWWWWRWWWKWRWS.................................................. RWWWWWWWWWSS.................................................... WRWWWSS......................................................... WWWWSS.......................................................... WRKWWWS......................................................... KWWW_CSS........................................................ WRWWWS.......................................................... _S.............................................................. S............................................................... ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 13:48:50 Current Time: Monday, 29-Apr-2002 16:35:42 EDT Restart Time: Monday, 29-Apr-2002 16:33:28 EDT Parent Server Generation: 0 Server uptime: 2 minutes 14 seconds Total accesses: 17491 - Total Traffic: 478.8 MB CPU Usage: u12.36 s67.61 cu0 cs0 - 59.7% CPU load 131 requests/sec - 3.6 MB/second - 28.0 kB/request 207 requests currently being processed, 134 idle workers _W_W__W______WW__W__W___WW____W_WW__W____WW___W_________________ W___WW____W_W_W____WW__W__W_____W_W____W_____________________W__ WRWCWWWCWW_WWWWW_CCWCWCSS....................................... W__W_W_W_W__W_W_W_____SS........................................ WCW_WWWWW_WKWCWWCSS............................................. ____WWW___WW____SS.............................................. WWWWKWWWWWWWWCWRWSS............................................. RWWWWWWWWWWWCWWWWCSS............................................ WWWWRRWWRCWWCWWCSS.............................................. WWW____W___SS................................................... RWWWWCRWWWSS.................................................... RWKWWWWWWSS..................................................... WWWWWWSS........................................................ WCWWWWCKSS...................................................... WCWWWWRSS....................................................... RWWCCSS......................................................... Now if all 16 of those processes start all the way up, with 64 threads each, we will have 1024 worker threads. Do we *really* need that many? Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 13:48:50 Current Time: Monday, 29-Apr-2002 16:36:32 EDT Restart Time: Monday, 29-Apr-2002 16:33:28 EDT Parent Server Generation: 0 Server uptime: 3 minutes 4 seconds Total accesses: 24094 - Total Traffic: 785.7 MB CPU Usage: u17.32 s105.95 cu0 cs0 - 67% CPU load 131 requests/sec - 4.3 MB/second - 33.4 kB/request 200 requests currently being processed, 82 idle workers GWGWGGWGGGGGGGGG.GWGGGGGGW.GGGGG.GGGGGGGGWWWG.GGGGGGWGGGGGGG.GGG WGGGGGGG.GWGWGWGGGGWGGGGGGGWGWGGWGGGGGGGGWGGGGGGGWGGWGGGGGGGGGGG W_WW__W_WW___WWWW_________W_SS.................................. WW_WWWWW__W_WW_W_W__W__W___SS................................... WWW__WWW_WW_WWW_CW___WSS........................................ WWWRWWW_WWW_WWWWWWW_RSS......................................... RWW__WW_W__W_W_WWW_W__SS........................................ WWWWWWWCW_WWWWWWWKWWWCCSS....................................... WWRWKWWCWWWCWRWWWWCWWSS......................................... W___W____W____WWS............................................... __WW_W_____W___SS............................................... _W_W_WW_W___W_SS................................................ CWCCWWCCCWWSS................................................... RWWCWWWWWCCRCSS................................................. CCCWCWWCWCCSSS.................................................. WWWWWWWWWRSS.................................................... I guess not. Above, we see the first churn happen at just over three minutes. It eventually got to the point where it had more than 192 idle workers so it started killing off processes. But we still have a long ways to go before all the workers for all the processes have been started... Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 13:48:50 Current Time: Monday, 29-Apr-2002 16:39:11 EDT Restart Time: Monday, 29-Apr-2002 16:33:28 EDT Parent Server Generation: 0 Server uptime: 5 minutes 43 seconds Total accesses: 42437 - Total Traffic: 1.7 GB CPU Usage: u12.41 s72.36 cu0 cs0 - 24.7% CPU load 124 requests/sec - 5.1 MB/second - 42.5 kB/request 381 requests currently being processed, 175 idle workers WGGGWGGGW..G.W.WWWGGGGGGWWWGGGWWGGWGWGGGWGWGGGGGGGGGGGGGGGGGGGGS GGGGG.GGGGGGGGGGGGGGG.GGGGGGGGGGWGGGGWGGG.GGGGGG................ _WW.WWW_WWW__WW___W__W___WWSSGGW..WGGGG.GGGGGGGG.G.W..GG........ _WW___W_____W___W_SSG..WW.GGWWG.GG.GGG.GWGG.G..GGGGGG.GG........ WW_W._______WW___W_W_WWWW______W_WW_SSGGG.WG.................... WW_WWWW___._________W_W_WSS..................................... _W_W_WWR_WRSSGGGWG.GGGGGGGGGGGGGGGGWGGGGWGGGWGGWGGGGGGGGGGGG.... RCWWWWWWRWWWCWWRCWWWWW.WWWWCCW___WW________SS.GWGGGG.GGGGGGG.... W_WW_.___W_________W____W____________SSGG....................... WWW.WWWWWWWWCWWWWWWWWWWWR_________SGGGGGGGGGGGGGGGGGGGG......... CCWWCWC.C.CRCCWCCWCCC..CC________W_W_W_SSGGGGG.GGGGGGG.......... WCWWWWWWCCWWWWW.WWCWCCW________W___SSGGGWGGGG..GGGGGGG.......... WWCKWCWWCWCWWRC.K____________SSGGGGGGGG.GGGGGGGGGG.............. ____WW__WW_____WW_______W________SS.....W....................... GG.GGGGGGWGGWWGGGGWGGGGGG.GGGGGGGGGGGGG.GGGGGGGGGGG............. WGGG.GWGG.G.GGGGGGGGG..GGGGGGG.GGGWGGGGGGGGGGG.G................ Now with almost 6 minutes under our belt (above), we see that Apache has indeed discovered that it started far too many processes and has begun cleaning up many of them up. I believe, because they die slowly, it kills too many... Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 13:48:50 Current Time: Monday, 29-Apr-2002 16:41:12 EDT Restart Time: Monday, 29-Apr-2002 16:33:28 EDT Parent Server Generation: 0 Server uptime: 7 minutes 44 seconds Total accesses: 54592 - Total Traffic: 2.4 GB CPU Usage: u16.24 s126.24 cu0 cs0 - 30.7% CPU load 118 requests/sec - 5.3 MB/second - 46.3 kB/request 10 requests currently being processed, 50 idle workers WGGG.GGG...G.W..W.GGGGGG.WWGGG..GG.G.GGG.GWGGGGGGGGGGGGGGGGGGGG. W__W________W___WWW_____W_____________W___________W________S.... W.W.WW.WW....WW..W....WWWW.WW.W..W.W.W......W....W.............. .W..WWW..WW.W...WW..WW..W..WW..W.W.............................. WGG..WGGGGGGWWWGGWG.GW.WWGGGWGG.G...GGGGGGGGGGGG.GGGGG.......... WGWG.GWGWW.WGWGGGGGWWG.G.GGWWGGGWGWWGGGWGGGGGGGGGGGG.GGG........ GGGWWWWWWWRWWGGGGG.GGGGGGGGGGGGGWGGGGG.GGGGGGG.GGGGG............ WWG.GG.GG.WWWGWWGGGGGG.G...GG.WGG..GGGGWGGGGGGG.GGGGG........... WG.WW.G..GGWWWGGGGG.G.GG.WGGGGWGGWWWGGGGGGGWGGGG................ .WG.GGWGWGG.GWGWGW.WWGWWGGGWGGWW.G.WGGGGGGGGGGGG................ GRGWWWW.G.GGGG.GGWWGG..GGGGGGGGGG.G.G.GGGGGWWGWWGGG............. WWG.W..WGWWG.W..WWWGWWGWWGGWGWG.WGWGWGGGGGGGGGGGGGGGGGGGG....... WW.WWGW.GGWGGGW.G.GWGGGGG.GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG...... GWWGW.GGW.GGWGGW.GGG.WGG.GGGGG.GGGGGGGGG.GGGGGG................. ................................................................ WGGG.G.GG.G.GGGGGGGGG..GGGGGGG.GGG.GGGGGGGGGGG.G................ ...and by 8 minutes (above) you can see that it is not in good shape. It is still responding to users, but the responses have continued to get slower in arriving. Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 13:48:50 Current Time: Monday, 29-Apr-2002 16:41:52 EDT Restart Time: Monday, 29-Apr-2002 16:33:28 EDT Parent Server Generation: 0 Server uptime: 8 minutes 24 seconds Total accesses: 57795 - Total Traffic: 2.6 GB CPU Usage: u9.62 s72.62 cu0 cs0 - 16.3% CPU load 115 requests/sec - 5.3 MB/second - 47.4 kB/request 233 requests currently being processed, 209 idle workers WWWWWWCCW_.__W__C______SSW................W..................... .__WRWW_WWW_WWWWWWWWW___WW_WWRWRW_C_R______S.................... WWWW.WRWWWWW______W_W____W_.SW.................................. W.WWWWWWC_W_W___WW___W__WSSWW................................... WWWWWWWWWWWWWWWR_W___W_.W___W_SS................................ WCCC__W__WWW_W_____WW_SGGWGWWWWGW..WGGGWGGGGG.GG.GGGWGGGG....... ____K_______W________________S.................................. .___________.________________SS................................. .CCWCCRCWRCC___________SS....................................... ____W_W_W___W__W___WW_SWW....................................... C.CW_W___W_______W_S.......................W..WW................ WWWWCCWCRW________.__SS......................................... _W___W__W____________S.......................................... WWWKW_W_W___W__W__SG.WGG.GGGGG.GGGGGGGGG.GGGGGG................. _WW___.______W_SS............................................... WGGG.G.GG.G.GGGGGGGGG..GGGGGGG.GGG.GGGGGGGGGGG.G................ Now that we realize we need more (above) we start the creation process again, but because we are sharing space with workers that are shutting down, we start more than we need to, now ending up with > 192 idle workers again (and likely to grow, due to more workers coming)... Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 13:48:50 Current Time: Monday, 29-Apr-2002 16:45:32 EDT Restart Time: Monday, 29-Apr-2002 16:33:28 EDT Parent Server Generation: 0 Server uptime: 12 minutes 4 seconds Total accesses: 78517 - Total Traffic: 3.9 GB CPU Usage: u22.21 s192.46 cu0 cs0 - 29.7% CPU load 108 requests/sec - 5.5 MB/second - 51.7 kB/request 210 requests currently being processed, 454 idle workers W__.W_W___.W_W__.________W__________________________SS.......... .GGWGGGGGGGG.GGGGGGGGGGG.GGWGGGGGGGGGGGGGGGGGGGGGGGGGGG......... __....W__.W.___W__._.____.___W_____WW___________________SS...... _.__..._WW.WWW_W.W___._WW__WWW_WW__W___________________SS....... ...__.___WWW..WW_.__W.W..___.___W__WWWWWW___________________SS.. W_WW_WW_R..._W_____..__W_.WWW__WW_W__SS.GGGGG.GG.GGG.GGGG....... ___W_W__________________W___W____W__W_W_______SS................ ________W_W_____WWW____W___W_________________W____________SS.... ._W____WW_____________WWW____WW_____W_______________SS.......... ____._.W.___.__.___..W_.WW_WW______________________SS........... _._.WW_W_.__WW_WW.W_W__WW_____________________SS................ ____WWW___.WW_WWW_WW_W_W_______________________SS............... _WW_W____________W_____WWW_W_____SS............................. _..W.__W.__W._WW_____._____W____SSGGGGGG.GGGGGG................. W..WWW.WWCWWW.WWWWWWWWWWWWWWWRWWWWWWWW_______SS................. WGGG.G.GG.G.GGGGGGGGG..GGGGGGG.GGG.GGGGGGGGGGG.G................ ...and sure enough, we've hit 454 idle workers. This patern is repeated over and over again during the whole hour of testing. I have more data if anyone wants to see it, but it roughly mimics the above. The hour was finished out with the following final status. Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 13:48:50 Current Time: Monday, 29-Apr-2002 17:36:31 EDT Restart Time: Monday, 29-Apr-2002 16:33:28 EDT Parent Server Generation: 0 Server uptime: 1 hour 3 minutes 3 seconds Total accesses: 320561 - Total Traffic: 20.7 GB CPU Usage: u14.51 s150.62 cu0 cs0 - 4.37% CPU load 84.7 requests/sec - 5.6 MB/second - 67.9 kB/request 357 requests currently being processed, 181 idle workers WWWWRWCWWRWSS....WG..GG..G..G..G....G...G.G..................... WWWR____WW_WWW_WR_W___W_W_SS.................................... _____W_._W_W____W___.W____SSGGGGGGGGGGGGGGGGGGGGGGGGGGG.GGGGGGG. W.WWCWWWWCWRSS.................................................. _W_W_._WW_.W___WWWW__WWW_W_SS.GGWGG.GGG.GGGWGG.GG.GG..GG.GG..... W_.__._R_WW___WW_W__WSSGG.G.GGG.G.GG.G.G.G.GGGGGGGGGGGGG........ RR__RW____WR__WW___W______SS.................................... WCWCCWWWWCCWWWCWCWCSS........................................... ____WW_____________W_W_W_W__W__W__W_________W______W__WW_W___SS. _____W__W___W_W_______SSGGGGGGGG.GGGGGG......................... __.______.__W._W.__W____SSGGGW.G.GGGGGGGGGGWGGG.WGGG.GGG.GG.GG.. ___CWWWCWW_WWWW_WWWWCWSS........................................ WWRWWWWWRWWWWWSS................................................ GWGGGGGGG.GGGGGGWGGG.GW.G.GGGGG.GWG.GGGGGGGGGGGGGGG.GGGGGGWGGGG. _W___W__W__W____W___________SS.................................. KWWWWW.WWWWWWCWWWWCCSS.GGGGGGGGGGGGGG.GGWG.GG.GGGGGG............ ============================================================================ ============================================================================ ============================================================================ ============================================================================ cvs-head + Bill's ============================================ Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 17:54:46 Current Time: Monday, 29-Apr-2002 18:09:59 EDT Restart Time: Monday, 29-Apr-2002 18:08:46 EDT Parent Server Generation: 0 Server uptime: 1 minute 13 seconds Total accesses: 4607 - Total Traffic: 124.9 MB CPU Usage: u3.22 s15.85 cu0 cs0 - 26.1% CPU load 63.1 requests/sec - 1.7 MB/second - 27.8 kB/request 117 requests currently being processed, 75 idle workers WWWWRW_WRWWWWR___WWWW_WW__WW_WWRWWWW_W_W___W_W_WWWR_WW_WWWRW____ WW___WWWWWW_WRWRWWWWW_W__WWWWWWWWWWWRWWWWWWWWWWW_WRWWR_WWWWRWWWW W________________W__________R_W__W__W____WWW__R_WWR_WWR___WW___W ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ Ramp up of rps comes a little slower for this one because there aren't as many processes starting threads in parallel... Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 17:54:46 Current Time: Monday, 29-Apr-2002 18:10:26 EDT Restart Time: Monday, 29-Apr-2002 18:08:46 EDT Parent Server Generation: 0 Server uptime: 1 minute 40 seconds Total accesses: 8351 - Total Traffic: 216.0 MB CPU Usage: u5.79 s31.74 cu0 cs0 - 37.5% CPU load 83.5 requests/sec - 2.2 MB/second - 26.5 kB/request 128 requests currently being processed, 128 idle workers WWWWWWWWWWWWWW__WWWWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWW__WWW_WWWWW_WW R___W__W_W__________W_________WW_W___W_____R_W___________WW____W WWWWWWWWW___WWWWWW_WW__WWWWWWW_WW___W_W_WWWWWWWWW__WWWWW_WWW__WW __W_____W________WW_____W___R_____W______R__RR__________________ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ But you can see that about a minute into handling the load spike (above) it is up to a respectable rps level (compared to the cvs-head only run) and has displayed no churn. Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 17:54:46 Current Time: Monday, 29-Apr-2002 18:12:02 EDT Restart Time: Monday, 29-Apr-2002 18:08:46 EDT Parent Server Generation: 0 Server uptime: 3 minutes 16 seconds Total accesses: 22300 - Total Traffic: 643.1 MB CPU Usage: u15.93 s87.42 cu0 cs0 - 52.7% CPU load 114 requests/sec - 3.3 MB/second - 29.5 kB/request 175 requests currently being processed, 81 idle workers __WW___W_WW_WW___WWW___WW__WWWWWWWWWWWWWWW__WWW_RWW_W_W_WWWWWW__ WWW_WW_WW_WWWW____WWW_WW__WWWWWW_WWW_WWWWWWW_WWWWW_W_WWWWWWWWWWW WWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW _W____WWW___W__W___W__W_W__WWWW_W__W_R______WW_____W______WW_W__ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ By the 3 minute mark (above) Apache is in full swing and has never had to kill off any processes. It started just what it needed to handle a nice smooth ramp up. Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 17:54:46 Current Time: Monday, 29-Apr-2002 18:12:35 EDT Restart Time: Monday, 29-Apr-2002 18:08:46 EDT Parent Server Generation: 0 Server uptime: 3 minutes 49 seconds Total accesses: 26517 - Total Traffic: 802.9 MB CPU Usage: u18.9 s106.51 cu0 cs0 - 54.8% CPU load 116 requests/sec - 3.5 MB/second - 31.0 kB/request 171 requests currently being processed, 149 idle workers ___WW_WWWW____W__WWW_WWWW_W_WWWWW__WWWWWWWWWWWW__W_WW_WWWW_W__W_ _WWWWWWWWWW_WWWWW_WWWWWWWWWWWWWWW_WW_WW_W_W__WWWWWWWWWWWWWWWW_WW W_WW___WWW__W__WWW_W_WW_WW_W_W__WW_____WWWW___WW__WW__WWWWW__W__ __WWWWWWWWW_WWWW___WWWWWWWWW_WW_WWW___W_W__WWW__W_WWWW_W_W_WWWW_ ________________________________________________________________ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ At about 3 1/2 minutes (above) it hits its peak. The number of long running responses has started to impact the rps. From this point on the rps will drop slightly (eventually down to about 95 rps) and the average request size will grow (to about 70 kB/request). It is starting to add another process worth of threads above. Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 17:54:46 Current Time: Monday, 29-Apr-2002 18:34:11 EDT Restart Time: Monday, 29-Apr-2002 18:08:46 EDT Parent Server Generation: 0 Server uptime: 25 minutes 25 seconds Total accesses: 151003 - Total Traffic: 9.3 GB CPU Usage: u116.53 s831.64 cu0 cs0 - 62.2% CPU load 99 requests/sec - 6.3 MB/second - 64.8 kB/request 224 requests currently being processed, 160 idle workers WW_WWW_W_WWWWWWW__WW_WWWWW_W_W_WWWW_W__WWW_WWW_W_W_WWW_WWW_WW_WW _WW_WWW_WWWWWWW_R_WWWWWWW_WW__WWWWWWWW_W_WWWWWWWWW__WW_WWWWWWWW_ ___W_W_W__WW___W_W____WW__W__W____WW_W_WW___W_W__W_____W__W__WWW WW_W_WW_W_WW___WW_W___WWWWW_WW___W____W___WW_W_WWW_W___W________ WWWWW_WWWWW_W_WWWWWWWW_W___W__WWW__WWWWWWW_WW_W_WWW__W_WWWWW_WW_ W_W_W_WWWW____W_W_____WW__WWWW_WW_W_WWW_WWW_W_W__WWW_W____WW____ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ By 25 minutes it has added two more processes to handle the workload backlog generated by the building number of long running responses. Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 17:54:46 Current Time: Monday, 29-Apr-2002 18:45:39 EDT Restart Time: Monday, 29-Apr-2002 18:08:46 EDT Parent Server Generation: 0 Server uptime: 36 minutes 53 seconds Total accesses: 209872 - Total Traffic: 13.6 GB CPU Usage: u162.62 s1233.66 cu0 cs0 - 63.1% CPU load 94.8 requests/sec - 6.3 MB/second - 68.0 kB/request 177 requests currently being processed, 143 idle workers GGGGWWWWWGWW.G..G.GWWG.W.W..GGGGGGGG.GGWG.G..G.WGWGGG.GGWW.WWGG. WWWWWW_WWW_WW_WW__WWWW_W_W_W_W_WWWWWW__W__WW_W_W__WWWWW_WWWW_WWW W_WW__W__W_W_WW__WW_W_____WW_WWWW_WW_W_WWWWWWW___WW___WWW_W_WW_W __WW_W__WW_____W_W______WW__W__WWWWW_WWW_WWW___WWWW_W_W_WW__WWW_ WW__WWW___WWW_W_WWW_WWW_W__WW_WW_WWW__WW___W_WWW__WW_WWWW_WWW_W_ ____W__W__WW__W_WWW_WWWW__W__WWW_WW_W_______W___W__WWW__W___W___ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ At about 35 minutes (above), enough of the long running responses cleared out so that it rose above 192 idle workers so it killed off one of the processes. Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 17:54:46 Current Time: Monday, 29-Apr-2002 18:48:49 EDT Restart Time: Monday, 29-Apr-2002 18:08:46 EDT Parent Server Generation: 0 Server uptime: 40 minutes 3 seconds Total accesses: 228745 - Total Traffic: 14.8 GB CPU Usage: u177.24 s1360.57 cu0 cs0 - 64% CPU load 95.2 requests/sec - 6.3 MB/second - 67.7 kB/request 187 requests currently being processed, 133 idle workers GGGGW....GW..G..G.G..G.W....GGGGGGGG.GG.G.G..G..G.GGG.GG.W..WGG. G..WG.W.G...WG.GWGWW.GGGGG...GGWWW.WW.GWGG.WG.GW.G...G.G.G..GWWG W__W_W___W____WW_WW__WW__W_W_W______WWWW__WWWW___WW__W_W_W___W_W WWWWWWW__WW___WWW_W___W__WWW_W__WW_WWWW_W____WWWWW_WWW_WW_WWWWW_ WW__WWWW__W__WW_W_W_WW_WWW__WWWW__WWWWW_WWW_WW_WWWWW_WW____WWW__ __W_WWWWWW____WWWWWWWWW_W__W_WWWW_W_W_WWW___WWW__W_W_W__WWWW_WWW _W___W_W_W_WWWW_____WWWW__W_W_WW_W_WWW_W_WW__W_WW_WW_WWWWWWWWWWW ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ By 40 minutes (above) the number of idle processes rose above 192 again so it killed off another process. This actually turned out to be slightly too many workers killed... Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 17:54:46 Current Time: Monday, 29-Apr-2002 18:59:18 EDT Restart Time: Monday, 29-Apr-2002 18:08:46 EDT Parent Server Generation: 0 Server uptime: 50 minutes 32 seconds Total accesses: 288883 - Total Traffic: 18.6 GB CPU Usage: u223.01 s1740.45 cu0 cs0 - 64.8% CPU load 95.3 requests/sec - 6.3 MB/second - 67.5 kB/request 214 requests currently being processed, 170 idle workers ................................................................ G..WG...G....G.G.G...GGGGG...GGW...WW.G.GG..G.G..G...G.G.G..G..G WWW_WW__WW__W_W_W_W_WW__W__W___WWW__WW_WWWW___W_W_WW_WWW___W__WW _W_W____WWW_WWW__WWW__W__WWW_WW__WWW_WW_W__W__W_WW_WW_WW_W_WW_WW WWWWWWWW_W_WWWWWW_WWWW_WWW_WW_W_WW__W__W_W_WWWWWW__W__W_WWW_WWW_ _W_W_W_WW_WWW_WWWW__W_WWW__WWW___WW_WW_WW__W____W__W__WW_WWW__WW _WW_WWWWWWW_W_WWWWW_W_WWWWW_W_WWWWWWW___WWW_____WW_WW__WWW_WWWWW _WW_W__W______WW___W_W_W__W__W____W_______WW_W__WW___WW_______WW ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ...so by the 50 minute mark it had started up another process. By this time (above) you can see that the first killed process has finished being cleaned up. Apache Server Status for www.apache.org Server Version: Apache/2.0.37-dev (Unix) Server Built: Apr 29 2002 17:54:46 Current Time: Monday, 29-Apr-2002 19:08:53 EDT Restart Time: Monday, 29-Apr-2002 18:08:46 EDT Parent Server Generation: 0 Server uptime: 1 hour 7 seconds Total accesses: 339622 - Total Traffic: 22.2 GB CPU Usage: u263.19 s2077.39 cu0 cs0 - 64.9% CPU load 94.2 requests/sec - 6.3 MB/second - 68.4 kB/request 219 requests currently being processed, 165 idle workers ................................................................ G..WG...G....G.G.G...GGGGG...GG....W..G.GG..G.G..G...G.G.G..G..G W_WWWWWWWWWWWWWWWWWWWWWWWWW_W_WWW_WWWWW_W_WWWWWWW_WW_WW_W_WW__WW W____WW__WWWWW___W_W_W__WW_WWWWW__WW_WW____W__W_W_WW__W___W__WW_ __WWWWWWWW_WW___RW_W_WW_W_W_WWWWW_WWW_WWWWWWWWWWWWWWWWW__WW__WWW W___WWW_W______WW_____W__W_WWWW_W___WW__W____W____________WW_W__ WWW_WWWWWW_WW_WW_WWW_WWWWWW_WW_WW__WWW_W_WWW_WW_WW_WWWWW_W_WWRWW WW___WW___W_________WW_W__W_W_____WWW___________WWW_WW______W_W_ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ And for an apples to apples comparisson, here is the status at the 1 hour mark (above). Requests per second has remained close to 100 the whole time. An extra 2 GB of data was sent, about 20,000 extra requests were processed with less system thrashing. -- Paul J. Reder ----------------------------------------------------------- "The strength of the Constitution lies entirely in the determination of each citizen to defend it. Only if every single citizen feels duty bound to do his share in this defense are the constitutional rights secure." -- Albert Einstein