httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alf H√łgemark <...@i100.no>
Subject Re: [users@httpd] Apache Hangs.. Server-Status shows all Reading
Date Sat, 10 Nov 2007 13:20:53 GMT
Hi

This seems to be quite similar to what I have seen, and others have also 
posted about the last couple of years.
I've done a lot of investigation, but have not found a real solution.
My problem is that the threads gets stuck in "W" / "Sending reply" state.

Are you using mod_php ? If so, it might be that one thread is doing 
work, while the other threads are waiting to acquire a lock on the php 
session file.
When PHP session files are used, the request are processed sequentially, 
and not in parallel for a user.

I suggest you also use "strace" unix command to see what the different 
threads are doing, that might be easier than using gdb.
When I use strace, I see that one thread is hanging on "poll" system 
call, until the timeout (Timeout Apache directive - default 300 seconds) 
occur. Then I see that the next thread hangs in "poll" and so on.
So if you set the "Timeout" lower, it might improve your situation.

You should also run "netstat", and see what the status of the sockets 
are. If they are almost all in CLOSE_WAIT, I think you are seeing 
something similar to me.

Here are some links to similar issues, I'm not sure if it will help you :
http://www.nabble.com/RE:-requests-time-out-under-load,-no-warnings-in-logs-p13065495.html
http://mail-archives.apache.org/mod_mbox/httpd-users/200611.mbox/%3c20061114084333.GC10809@nexus.intra.coretrek.com%3e

( and the other posts to that thread)
http://www.linuxforums.org/forum/servers/27519-alright-im-dying-here-apache-help-mysql-php.html
http://osdir.com/ml/apache.mod-ssl.user/2004-10/msg00009.html

I have also seen other posts from people with very similar problems, 
lots of thread stuck in a state.

What is your environment ? I.e. what version of Apache, what operating 
system and version, are you generating dynamic pages, and if so, what 
tools (for example php) are you using ?
Do you have a firewall or a load balancer in front of your web server ?

Are almost all the requests from the same IP adresse ? Are almost all 
the request for the same URL ?

What my theory is at the moment, after throwing away some other 
theories, are that somehow a client creates several hundred requests for 
the same URL.
Since in my case this happens fairly often, several times a day on a 
uncontroversial high traffic site, from different users, I do not think 
it is someone doing a DoS attack.

I think it might be a problem with a firewall or a proxy or maybe the 
browser (I have seen it happen for people using both Windows and Mac, 
both Firefox and IE), that sits between the web server and the end user. 
Somehow the closing of the socket/connection by the browser is not 
picked up by the web server. And somehow the browser issues lots of 
requests.
I have not completely ruled out that it might be a Apache web server bug 
as well.
But since I have no way of reproducing this, apart from observing it 
happening quite often, it is very hard to get any further.
So any help would be greatly appreciated.

Regards
Alf Hogemark



Sander Temme wrote:
>
> On Nov 7, 2007, at 9:31 AM, Andrew Rosolino wrote:
>
>> I restarted apache and all was fine.. but then 20 minutes later they 
>> went
>> back all into a reading state.. it appears as if slowly each 
>> processes goes
>> into the reading state?? I dont understand what the problem is.
>
> If you have gdb on the box (and have some acumen using it), you can 
> attach to one of the processes and see if it is hanging in something 
> obvious.
>
> If not, and your traffic is plaintext, I recommend the tcpdump 
> approach to see what is going on traffic-wise, and what triggers the 
> issue.  I'd dump the traffic to disk (tcpdump -i ethX -s 0 -w 
> /wherever/traffic.dump port 80) and take it over to a box with 
> Ethereal/Wireshark for some cozy analysis.
>
> If you're being DOS attacked by trickle requests, you could try 
> setting a very low timeout (default is 5 minutes which doesn't seem to 
> be working for you) and perhaps use mod_evasive or somesuch to flag 
> and firewall the bad clients.
>
> S.
>

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message