httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stuart Haas <sh...@vibespin.com>
Subject Re: [users@httpd] Server Load Issue
Date Wed, 16 Jun 2004 18:47:06 GMT
I don't don't think it's a DOS.  The W's appear as a result of the load, 
not as the cause of it.  I was watching as the load build and everything 
went to W.  Hostnamelookups are off.

Server status with a normal load:

Server Version: Apache
Server Built: Mar 1 2004 12:14:32

------------------------------------------------------------------------

Current Time: Wednesday, 16-Jun-2004 14:41:38 EDT
Restart Time: Tuesday, 15-Jun-2004 14:56:46 EDT
Parent Server Generation: 0
Server uptime: 23 hours 44 minutes 51 seconds
Total accesses: 740073 - Total Traffic: 6.7 GB
CPU Usage: u1034.06 s33.86 cu2.35 cs1.93 - 1.25% CPU load
8.66 requests/sec - 81.8 kB/second - 9.5 kB/request
34 requests currently being processed, 30 idle workers

WK_K__KK___W_KK______KK_WW_KK___.K_CKK_KKK_K__K_W_WK.W_K_KK.W..W
......_...................K._...................................
..............._.................................W..............
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
........................................................



Vasiliy Boulytchev wrote:

>Ok, Please tell me if I am wrong, but "W" state is "sending reply"  hmmmmmm.
>DOS attack?   Does your web server do DNS queries?  If so, turn them
>off.......  Anyone else?  
>
>I assume no other procs are thrashing the system.........
>
>
>Vasiliy Boulytchev
>Colorado Information Technologies, Inc.
>http://www.coinfotech.com
>
>-----Original Message-----
>From: Stuart Haas [mailto:shaas@vibespin.com] 
>Sent: Wednesday, June 16, 2004 12:25 PM
>To: users@httpd.apache.org
>Subject: Re: [users@httpd] Server Load Issue
>
>
>Ok,
>
>This is what we get: (server status and a whole lot more...):
>
>Server Version: Apache
>Server Built: Mar 1 2004 12:14:32
>
>Current Time: Wednesday, 16-Jun-2004 14:12:27 EDT Restart Time: Tuesday,
>15-Jun-2004 14:56:46 EDT Parent Server Generation: 0 Server uptime: 23 hours
>15 minutes 41 seconds Total accesses: 717294 - Total Traffic: 6.5 GB CPU
>Usage: u382.63 s15.45 cu1.57 cs1.27 - .479% CPU load
>8.57 requests/sec - 80.9 kB/second - 9.4 kB/request
>308 requests currently being processed, 8 idle workers
>
>WWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWW
>WWWWWW_WWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
>WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
>WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWW
>WWRWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWW_WWWWWWWWWKWW_WWWWWW_WWWW....
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>........................................................
>
>
>---
>
>( I think this is when things are cooling down - CPU hits 100% on all 4 when
>peaking)
>
>Top:
>14:12:30 up 27 days, 13:00, 4 users, load average: 261.00, 133.84, 55.70
>391 processes: 370 sleeping, 19 running, 2 zombie, 0 stopped CPU states: cpu
>user nice system irq softirq iowait idle total 42.7% 0.0% 3.5% 0.1% 0.5%
>52.9% 0.0% cpu00 37.6% 0.0% 4.0% 0.0% 1.1% 57.1% 0.0%
>cpu01 35.2% 0.0% 1.7% 0.6% 0.0% 62.3% 0.0%
>cpu02 61.8% 0.0% 4.9% 0.0% 0.8% 32.3% 0.0%
>cpu03 36.1% 0.0% 3.5% 0.0% 0.3% 59.9% 0.0%
>Mem: 4125972k av, 4105812k used, 20160k free, 0k shrd, 3848k buff 3213112k
>actv, 612800k in_d, 63032k in_c
>Swap: 4096020k av, 867116k used, 3228904k free 865776k cached
>
>---
>
>iostat 5
>
>avg-cpu: %user %nice %sys %idle
>5.45 0.00 0.24 94.32
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 4.36 22.42 106.24
>52403004 248317390
>
>avg-cpu: %user %nice %sys %idle
>79.79 0.00 20.21 0.00
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 47.83 347.83
>1565.22 80 360
>
>avg-cpu: %user %nice %sys %idle
>95.02 0.00 4.98 0.00
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 38.00 688.00
>272.00 344 136
>
>avg-cpu: %user %nice %sys %idle
>59.15 0.00 40.85 0.00
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 145.71 937.14
>1165.71 328 408
>
>avg-cpu: %user %nice %sys %idle
>94.86 0.00 5.14 0.00
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 15.87 0.00 660.32
>0 416
>
>avg-cpu: %user %nice %sys %idle
>96.95 0.00 3.05 0.00
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 71.23 0.00 898.63
>0 656
>
>avg-cpu: %user %nice %sys %idle
>92.92 0.00 7.08 0.00
>---
>
>vmstat 5
>procs memory swap io system cpu
>r b swpd free buff cache si so bi bo in cs us sy id wa 0 328 1019384 23668
>3380 725624 3 0 4 0 2 2 1 0 2 2 0 328 1019828 24896 3396 725500 442 134 442
>150 277 493 2 1 0 97 0 330 1019468 29936 3412 725528 466 0 466 11 282 326 2
>0 0 98 0 324 1019660 27316 3444 725656 478 124 483 131 298 359 3 1 0 96 0
>320 1015860 70088 3460 725676 428 308 429 330 308 367 3 1 0 96 0 323 1015500
>71708 3476 725696 453 0 453 13 452 521 15 1 0 84 0 326 1014520 90796 3488
>725716 413 54 414 119 413 468 13 1 0 86
>11 289 1012908 75644 3524 725760 517 0 522 18 1221 692 61 4 0 35 0 304
>1012472 76696 3540 725856 430 142 444 199 881 684 49 2 0 49 0 302 1012052
>78696 3568 725880 438 0 438 50 393 411 9 0 0 91
>64 176 1005584 72120 3852 725972 478 46 537 78 1540 1063 69 4 0 27
>---
>
>pstree
>init-T-atd
>+-bdflush
>+-cache_clean
>+-crond---crond---cache_clean
>+-gpm
>+-httpd-T-305*[httpd]
>| L-4*[httpd---links]
>+-irqbalance
>+-keventd
>+-khubd
>+-3*[kjournald]
>+-klogd
>+-kscand
>+-ksoftirqd/0
>+-ksoftirqd/1
>+-ksoftirqd/2
>+-ksoftirqd/3
>+-kswapd
>+-kupdated
>+-mdadm
>+-mdmpd
>+-mdrecoveryd
>+-6*[mingetty]
>+-ntpd
>+-3*[raid1d]
>+-rhnsd
>+-2*[sendmail]
>+-sldfs_rreq
>+-sldrm_coll---sldrm_hist
>+-sldtkd
>+-sshd-T-sshd---sshd
>| +-sshd---sshd---bash---top
>| +-sshd---sshd---bash---su---bash---top
>| +-sshd---sshd---bash
>| L-sshd---sshd---bash---su---bash---pstree
>+-syslogd
>L-xinetd
>
>Vasiliy Boulytchev wrote:
>
>  
>
>>During the slowdown, what does /server-status/ show? 
>>
>>
>>Vasiliy Boulytchev
>>Colorado Information Technologies, Inc.
>>http://www.coinfotech.com
>>
>>-----Original Message-----
>>From: Stuart Haas [mailto:shaas@vibespin.com]
>>Sent: Wednesday, June 16, 2004 10:50 AM
>>To: users@httpd.apache.org
>>Subject: [users@httpd] Server Load Issue
>>
>>
>>Hi,
>>
>>We are having a strange problem with Apache where the server load 
>>skyrockets and leaves our site very unresponsive.  The load average on 
>>the server typically is well under 0.5.  Once a threshold is reached (I 
>>can't pin this down exactly, but we usually reach it several times over 
>>our busy hours), the load immediately jumps to around 150 and more.
>>After a while, every thing returns to normal.  I would expect that the 
>>server load would increase linearly as traffic increases, but that is
>>not the case.   Out site is not that busy as we serve about 30,000 pages 
>>a day.
>>
>>We are running apache 2.0.46 (Redhat's) on Redhat Enterprise Linux 3.  
>>The hardware is  2x 3.06Ghz Xeon with 4GB RAM, so I don't believe the 
>>box is underpowered.  Our content management system is PHP based and in 
>>an effort to solve this issue went to the Zend performance suite, but 
>>this has not had any effect on this problem.  There are no error messages
>>    
>>
>in any of the logs.
>  
>
>>How would we go about troubleshooting this issue?
>>
>>Stuart
>>
>>
>>
>>---------------------------------------------------------------------
>>The official User-To-User support forum of the Apache HTTP Server Project.
>>See <URL:http://httpd.apache.org/userslist.html> for more info.
>>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>>  "   from the digest: users-digest-unsubscribe@httpd.apache.org
>>For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>>
>>---------------------------------------------------------------------
>>The official User-To-User support forum of the Apache HTTP Server Project.
>>See <URL:http://httpd.apache.org/userslist.html> for more info.
>>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>>  "   from the digest: users-digest-unsubscribe@httpd.apache.org
>>For additional commands, e-mail: users-help@httpd.apache.org
>>
>> 
>>
>>    
>>
>
>--
>------------------------------
>Stuart Haas
>IT Director
>Vibe/Spin Ventures
>---
>Assistant Director of Technology
>Miller Publishing Group
>212.448.7493 
>
>
>
>---------------------------------------------------------------------
>The official User-To-User support forum of the Apache HTTP Server Project.
>See <URL:http://httpd.apache.org/userslist.html> for more info.
>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>   "   from the digest: users-digest-unsubscribe@httpd.apache.org
>For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
>---------------------------------------------------------------------
>The official User-To-User support forum of the Apache HTTP Server Project.
>See <URL:http://httpd.apache.org/userslist.html> for more info.
>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>   "   from the digest: users-digest-unsubscribe@httpd.apache.org
>For additional commands, e-mail: users-help@httpd.apache.org
>
>  
>

-- 
------------------------------
Stuart Haas
IT Director
Vibe/Spin Ventures
---
Assistant Director of Technology
Miller Publishing Group
212.448.7493 



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message