httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Rosolino <and...@shiftcode.com>
Subject Re: [users@httpd] Apache Hangs.. Server-Status shows all Reading
Date Sun, 11 Nov 2007 16:49:07 GMT

Im using this for php..
LoadModule php4_module modules/libphp4.so

not sure if that is mod_php or what

I tried using strace but my server was going extremly slow when I started it
probably because we get so much traffic so i couldnt debug it...

I did run the gdb on a process and got this.. 
#0  0x003307a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 
#1  0x00408dbd in poll () from /lib/tls/libc.so.6 
#2  0x00af595f in apr_poll (aprset=0xbfe55310, num=1, nsds=0xbfe5530c, 
    timeout=300000) at poll.c:130 
#3  0x00af5d5f in apr_wait_for_io_or_timeout (f=0x0, s=0x959b4a8,
for_read=1) 
    at waitio.c:54 
#4  0x00aec19c in apr_socket_recv (sock=0x959b4a8, 
    buf=0x9624688 "\n# AddHandler application/x-httpd-php .php
.sc\r\nIndexIgnore .htaccess */.??* *~ *# */HEADER* */README*
*/_vti*\r\nDirectoryIndex home.gc index.gc index.php\r\n\r\n<Limit GET
POST>\r\n#The next line modified b"..., 
    len=0xbfe553e8) at sendrecv.c:87 
#5  0x00fcc0dc in socket_bucket_read (a=0x95c0468, str=0xbfe553e4, 
    len=0xbfe553e8, block=APR_BLOCK_READ) at apr_buckets_socket.c:36 
#6  0x00fcd06a in apr_brigade_split_line (bbOut=0x95d0f50, bbIn=0x959b9a0, 
    block=APR_BLOCK_READ, maxbytes=8192) at apr_brigade.c:289 
#7  0x080b9a9b in core_input_filter (f=0x959b968, b=0x95d0f50, 
    mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core.c:3802 
#8  0x0806e8db in logio_in_filter (f=0x959b928, bb=0x95d0f50, 
    mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at
mod_logio.c:115 
#9  0x080b3ae3 in ap_rgetline_core (s=0x95d02a0, n=8192, read=0xbfe55528, 
    r=0x95d0288, fold=0, bb=0x95d0f50) at protocol.c:230 
#10 0x080b4577 in ap_read_request (conn=0x959b580) at protocol.c:586 
#11 0x0808bc6e in ap_process_http_connection (c=0x959b580) at
http_core.c:271 
---Type <return> to continue, or q <return> to quit--- 
#12 0x080b0a62 in ap_run_process_connection (c=0x959b580) at connection.c:43 
#13 0x080a5ad1 in child_main (child_num_arg=Variable "child_num_arg" is not
available. 
) at prefork.c:610 
#14 0x080a5cfb in make_child (s=Variable "s" is not available. 
) at prefork.c:704 
#15 0x080a5d8c in startup_children (number_to_start=5) at prefork.c:722 
#16 0x080a645f in ap_mpm_run (_pconf=0x8c2c0a8, plog=0x8c6c1a8, s=0x8c32ed0) 
    at prefork.c:941 
#17 0x080ab803 in main (argc=4, argv=0xbfe558c4) at main.c:638 

havent tried netstat yet because the problem isnt occuring right now..
Here is some info on the Apache we are using:
Server Version: Apache/2.0.61 (Unix) mod_ssl/2.0.61 OpenSSL/0.9.7a
mod_auth_passthrough/2.1 FrontPage/5.0.2.2635 mod_bwlimited/1.4 

We are running REDHAT Enterprise 4 i686 almost all our pages are dynamic
php.. and its running about 200 websites.. we dont have a load balancer.. 

All the requests are not from the same ip address very random.


Alf H√łgemark wrote:
> 
> Hi
> 
> This seems to be quite similar to what I have seen, and others have also 
> posted about the last couple of years.
> I've done a lot of investigation, but have not found a real solution.
> My problem is that the threads gets stuck in "W" / "Sending reply" state.
> 
> Are you using mod_php ? If so, it might be that one thread is doing 
> work, while the other threads are waiting to acquire a lock on the php 
> session file.
> When PHP session files are used, the request are processed sequentially, 
> and not in parallel for a user.
> 
> I suggest you also use "strace" unix command to see what the different 
> threads are doing, that might be easier than using gdb.
> When I use strace, I see that one thread is hanging on "poll" system 
> call, until the timeout (Timeout Apache directive - default 300 seconds) 
> occur. Then I see that the next thread hangs in "poll" and so on.
> So if you set the "Timeout" lower, it might improve your situation.
> 
> You should also run "netstat", and see what the status of the sockets 
> are. If they are almost all in CLOSE_WAIT, I think you are seeing 
> something similar to me.
> 
> Here are some links to similar issues, I'm not sure if it will help you :
> http://www.nabble.com/RE:-requests-time-out-under-load,-no-warnings-in-logs-p13065495.html
> http://mail-archives.apache.org/mod_mbox/httpd-users/200611.mbox/%3c20061114084333.GC10809@nexus.intra.coretrek.com%3e

> ( and the other posts to that thread)
> http://www.linuxforums.org/forum/servers/27519-alright-im-dying-here-apache-help-mysql-php.html
> http://osdir.com/ml/apache.mod-ssl.user/2004-10/msg00009.html
> 
> I have also seen other posts from people with very similar problems, 
> lots of thread stuck in a state.
> 
> What is your environment ? I.e. what version of Apache, what operating 
> system and version, are you generating dynamic pages, and if so, what 
> tools (for example php) are you using ?
> Do you have a firewall or a load balancer in front of your web server ?
> 
> Are almost all the requests from the same IP adresse ? Are almost all 
> the request for the same URL ?
> 
> What my theory is at the moment, after throwing away some other 
> theories, are that somehow a client creates several hundred requests for 
> the same URL.
> Since in my case this happens fairly often, several times a day on a 
> uncontroversial high traffic site, from different users, I do not think 
> it is someone doing a DoS attack.
> 
> I think it might be a problem with a firewall or a proxy or maybe the 
> browser (I have seen it happen for people using both Windows and Mac, 
> both Firefox and IE), that sits between the web server and the end user. 
> Somehow the closing of the socket/connection by the browser is not 
> picked up by the web server. And somehow the browser issues lots of 
> requests.
> I have not completely ruled out that it might be a Apache web server bug 
> as well.
> But since I have no way of reproducing this, apart from observing it 
> happening quite often, it is very hard to get any further.
> So any help would be greatly appreciated.
> 
> Regards
> Alf Hogemark
> 
> 
> 
> Sander Temme wrote:
>>
>> On Nov 7, 2007, at 9:31 AM, Andrew Rosolino wrote:
>>
>>> I restarted apache and all was fine.. but then 20 minutes later they 
>>> went
>>> back all into a reading state.. it appears as if slowly each 
>>> processes goes
>>> into the reading state?? I dont understand what the problem is.
>>
>> If you have gdb on the box (and have some acumen using it), you can 
>> attach to one of the processes and see if it is hanging in something 
>> obvious.
>>
>> If not, and your traffic is plaintext, I recommend the tcpdump 
>> approach to see what is going on traffic-wise, and what triggers the 
>> issue.  I'd dump the traffic to disk (tcpdump -i ethX -s 0 -w 
>> /wherever/traffic.dump port 80) and take it over to a box with 
>> Ethereal/Wireshark for some cozy analysis.
>>
>> If you're being DOS attacked by trickle requests, you could try 
>> setting a very low timeout (default is 5 minutes which doesn't seem to 
>> be working for you) and perhaps use mod_evasive or somesuch to flag 
>> and firewall the bad clients.
>>
>> S.
>>
> 
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Apache-Hangs..-Server-Status-shows-all-Reading-tf4766110.html#a13693182
Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message