Mailing-List: contact users-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@httpd.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <4CF57E4F.7090202@adaptr.nl>
Date: Tue, 30 Nov 2010 23:44:31 +0100
From: Jeroen Geilman <jeroen@adaptr.nl>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.1.15) Gecko/20101113 Mnenhy/0.8.3 Thunderbird/3.0.10
 ThunderBrowse/3.3.4
MIME-Version: 1.0
To: users@httpd.apache.org
References: <AANLkTikT8j8KhrZ8hCi0zrtAQKLJZgwmfN+cvCKpmBAp@mail.gmail.com>
In-Reply-To: <AANLkTikT8j8KhrZ8hCi0zrtAQKLJZgwmfN+cvCKpmBAp@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [users@httpd] Connection Issues

On 11/29/2010 11:25 PM, Travis Whitton wrote:
> Hi,
>
> We're experiencing some odd behavior regarding connections taking a
> long time to establish to our website. We've been running Apache in
> production for over three years now and have recently began
> experiencing issues where the server-status page, static, and dynamic
> content response times will slow anywhere from a few seconds to long
> enough for the connection to timeout.
>
> Initially thinking we might be hitting some hard limits with the OS,
> we've thoroughly audited our sysctl variables, tried disabling
> iptables and conntrack, and ensured that we're not running out of
> ephemeral ports or anything along those lines. Looking at netstat, it
> seems we have a pretty large number of connections in TIME_WAIT which
> is understandable since this is a high traffic website, but I'm
> wondering if this value could indicate we're backlogging on TCP
> connections or something along those lines?
>
> [root@RHL073 ipv4]# netstat -an | awk '/^tcp/ {A[$(NF)]++} END {for (I
> in A) {printf "%5d %s\n", A[I], I}}'
> 34723 TIME_WAIT
>      3 CLOSE_WAIT
>    275 FIN_WAIT1
>     74 FIN_WAIT2
>   8824 ESTABLISHED
>    815 SYN_RECV
>    102 CLOSING
>     30 LAST_ACK
>     10 LISTEN
>
> In an effort to tune things, I've tried playing with the TCP timeout
> settings a bit, and the response times have improved somewhat. Please
> note that I've been testing response times using the loopback
> interface to rule out any ethernet hardware issues.
>
> echo 15>  /proc/sys/net/ipv4/tcp_fin_timeout
> echo 1>  /proc/sys/net/ipv4/tcp_tw_recycle
> echo 1>  /proc/sys/net/ipv4/tcp_tw_reuse
>
> We're running prefork, and have configured the client settings to what
> seem to be reasonable limits for our hardware.
>
> <IfModule prefork.c>
> StartServers       100
> MinSpareServers    100
> MaxSpareServers   200
> ServerLimit       1500
> MaxClients        1500
> MaxRequestsPerChild 1000000
> </IfModule>
>
>    

Forking new children is VERY expensive, compared to the alternatives.

If 1500 concurrent clients is common for your site, consider starting up 
that many as well.
min/maxspare is only meant to handle bursts, not define your normal load.
Your settings mean "accept up to 1500 concurrent connections, but only 
RUN 300 threads when you don't have that many clients"

Since apache will have to fork up to 1200 threads in rapid succession 
when the load spikes, this will cause startup throttling after only a 
few seconds, which is causing your timeouts.

You should change these to AT LEAST 1000 startup, 100 minspare and 200 
maxspare - if 1500 is your actual max load, and not a limit you imposed 
because you think it can't handle more.
It can handle many more, if you have the memory for them.

With 1500 concurrent connections, I would long ago have moved to worker 
combined with proxying dynamic content to a separate prefork instance.
This will optimize memory and resource usage to such an extent that you 
can easily run 5000 clients concurrently.

Worker threads are much more efficient and take far less memory than 
prefork children, therefore they suffer far less from being short-lived 
(due to low maxrequest settings)

Unless the majority of these requests are for dynamic content (they 
rarely are), I predict you can increase performance several fold.

-- 
J.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org