httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sander Temme <scte...@apache.org>
Subject Re: [users@httpd] Apache 2.0.58 + Solaris 5.9: status "...reading..." & TCP state "FIN_WAIT_2"
Date Fri, 26 Jan 2007 17:18:46 GMT
Chirouze,

On Jan 26, 2007, at 8:41 AM, Chirouze Olivier wrote:

> Thanks for your prompt reply. We will definitively upgrade soon, just
> "to be up to date"... But because nothing is said about that point in
> docs I'm wondering if that will make any difference...

I doubt it. The Apache 2.0.x branch at this moment mainly gets  
security fixes, and not a lot else.  The difference between  
subsequent versions tends to be fairly small, and we tend to tell you  
what the fixes are.

You tell us that you are seeing an unusually high number of children  
sitting in 'R' stage, which means they are attempting to read a full  
request.  Reading requests *should* not take very long (this also  
depends on the kind of requests you expect to get), and it may be  
that you are under attack.  This may be deliberate or accidental.

Do you have something like a Firewall in front of your server that  
drops idle TCP connections?  If this happens, your Apache server will  
not notice this and keep trying to read the incoming data.

In any case, you may try to reduce the value of the Timeout directive  
in your configuration file.

The high number of FIN_WAIT_2 connections you observe may be normal,  
but it may be the result of the situation sketched above.  Once  
Apache is done with a connection, it'll close that and forget about  
it.  The Apache child will then get ready to accept a new connection,  
and the kernel takes care of closing the TCP connection.  It does  
this by sending a FIN packet to the client, who sends an ACK back.   
Then the client is supposed to send its own FIN, and while this  
happens the server will have that socket in FIN_WAIT_2 state.  After  
the client sends its FIN, the server ACKnowledges and puts the socket  
into TIME_WAIT before it can be re-used.  You can see a full diagram  
of this process in Figure 18.12 in TCP Illustrated, Volume 1 by W.  
Richard Stevens.

If you are dealing with clients that don't bother sending their FIN,  
or have something in-between that messes with the connection and its  
termination, you may be stuck with this.  You can use kernel tunables  
like the ones you mention below to reduce the overhead.

Once again, Apache has nothing to do with the TCP handshake process.   
However, you may be able to reduce the effect of these 'R'  
connections by reducing the timeout and raising the number of  
available child processes (if your log file tells you that the server  
reached MaxClients).

Regards,

Sander

> Anyway, we're providing both HTTP and HTTPS. Might be interesting  
> to try
> recognize if this happens on both? I will have a look at it.
>
> Do you think you might give me the values of the following Unix params
> on your Solaris 9 installs?
>
> tcp_fin_wait_2_flush_interval
> tcp_keepalive_interval
> tcp_ip_abort_interval
>
> Thanks in advance,
>
> Olivier
>
> Olivier CHIROUZE
> I&0 Infrastructure
> Volvo Information Technology
>
> -----Original Message-----
> From: Richard de Vries [mailto:richard_devries@yahoo.com]
> Sent: 26 January 2007 17:35
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] Apache 2.0.58 + Solaris 5.9: status
> "...reading..." & TCP state "FIN_WAIT_2"
>
> Interesting problem.
>
> I am running Apache 2.0.59 as a reverse proxy on multiple Solaris 9  
> and
> AIX servers and have never encountered these types of issues. Perhaps
> you should try upgrading to 2.0.59 on one of your development machines
> and see if that makes a difference. If not, it is most likely an OS
> and/or configuration issue.
>
> What other plugins are you running? Also, is this HTTP proxying, or
> HTTPS?
>
> ----- Original Message ----
> From: Chirouze Olivier <olivier.chirouze@volvo.com>
> To: users@httpd.apache.org
> Sent: Friday, January 26, 2007 9:56:46 AM
> Subject: [users@httpd] Apache 2.0.58 + Solaris 5.9: status
> "...reading..." & TCP state "FIN_WAIT_2"
>
>
> Hi all,
>
> I'm facing a quite tricky situation with Apache 2.0.58 running on
> Solaris 5.9.
>
> Apache is running as a reverse proxy (mod_proxy + mod_rewrite).
> The maximum concurrent connections is set to 150.
>
> Because we reached the maximum a few times and got the reverse proxy
> saturated, we started monitoring the Apache status page (/status).
> We noticed that many requests were in the "..reading.." state (up to
> 40!), and they block a lot of slots.
>
> At first, we upgraded from 2.0.47 to 2.0.58 because it seemed there  
> was
> a security hole in the earlier, fixed in 2.0.48.
> I found some explanation here:
> http://www.monkeybrains.net/~rudy/example/server_busy_state.html.
>
> The thing is, the situation is starting to appear again with 2.0.58.
>
> We've gone down to Unix and found that most of these requests were in
> "FIN_WAIT_2" TCP state, and for a while (approx. 8min!!).
>
> We found this: http://httpd.apache.org/docs/2.0/misc/fin_wait_2.html.
> What it says, in a word, is that these things can happen and are
> "normal": the connection stays in "FIN_WAIT_2" state until the  
> timeout,
> if clients do not close it properly. They just say it can be a problem
> on the Unix point of view because.
> I don't know if this is still true for 2.0 because the article was  
> just
> copied from 1.3. The thing is, it says that "The connections in
> FIN_WAIT_2 do not tie up an httpd process". For us, IT DOES! Every
> "..reading.." request happend to be in the "FIN_WAIT_2" state.
>
> We have contacted Sun to get their opinion. The short answer is  
> "you can
> change the FIN_WAIT_2 timeout but be careful because wrong tuning will
> have negative impact. Maybe you should wonder why these connections  
> stay
> alive". As far as I understood, the connection is not closed by the
> client. The server (Apache) does nothing wrong. But maybe it does,  
> as it
> doesn't leave the process free?
>
> My questions are:
> Does anyone have heard about similar problems?
> Why do these connections hold a process of Apache while the
> documentation says it doesn't?
> Do you recon tuning the Unix timeout would help? (current value of
> tcp_fin_wait_2_flush_interval: 675000 ms - 11min!! This looks just
> huge!)
>
> Thanks in advance,
>
> Olivier
>
> Olivier CHIROUZE
> I&0 Infrastructure
>
> Volvo Information Technology
>
>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
> ______________________________________________________________________ 
> __
> ____________
> Do you Yahoo!?
> Everyone is raving about the all-new Yahoo! Mail beta.
> http://new.mail.yahoo.com
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server  
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>



-- 
sctemme@apache.org            http://www.temme.net/sander/
Open Source Software Consultant
PGP FP: 51B4 8727 466A 0BC3 69F4  B7B8 B2BE BC40 1529 24AF




Mime
View raw message