Mailing-List: contact users-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@httpd.apache.org
Received-SPF: pass (athena.apache.org: domain of trevor.phillips@gmail.com
 designates 209.85.200.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=McJ0iElYPqKFglZBtdfeDUhfCjy+PBMf/plqm6/y1cAPqPeeopASAw+fNI26T+bbne
         JaREM4iEgZTjEO+b2pgKV4XfBWEw7pLdJPICp3yo3eEdrjjwaKY3e4Xx/keUv5OqnYzh
         OZiB36oguq+b6XHfuDIXbDmqA877NXkO0WY38=
MIME-Version: 1.0
Date: Wed, 4 Mar 2009 15:39:44 +0900
Message-ID: <3fff79a80903032239m386dbb44y3ee79998ab67414@mail.gmail.com>
From: Trevor Phillips <trevor.phillips@gmail.com>
To: users@httpd.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Subject: [users@httpd] Apache bottleneck problems

Hi!
I've been a long time administrator of Apache servers in a University
environment (translation: a LOT of variety in what content & apps are
on the servers), and our configuration isn't getting simpler.

One of our sites is currently set up with a reverse proxy in front of
two back-end sites, of which one site is load balanced across two
servers. One of the goals with the load-balanced section of the site
was for fail-over if something bad happened to one of the servers.

The problem we're having is that if "something bad" happens to any
back-end server, then the reverse proxy workers are all soaked up by
incoming requests for the broken part, which then blocks access to the
rest of the content.

Is there a way to limit how many workers buffer up requests for each
back-end server? Or even based on Location or URL pattern match?

Even within one server, we encounter this problem on a smaller scale
occasionally - one heavy dynamic part of the site grinds to a crawl,
all apache daemons queue up queries for that part, and all the rest of
the content suffers as a consequence.  If a sub-part of the site does
encounter problems, then it would be preferable to politely drop extra
connections to that part of the site and get on with providing the
rest of the content to clients.

Here's snippets of part of our apache & proxy config:

  Timeout 300
  KeepAlive On
  MaxKeepAliveRequests 100
  KeepAliveTimeout 15

  <IfModule mpm_prefork_module>
    ServerLimit           700
    StartServers          20
    MinSpareServers       10
    MaxSpareServers      30
    MaxClients          700
    MaxRequestsPerChild 500
  </IfModule>

  <Proxy balancer://backend>
         BalancerMember  http://10.50.4.6
         BalancerMember  http://10.50.4.7
         ProxySet lbmethod=byrequests
  </Proxy>

  RewriteRule ^/$ balancer://backend/ [P,L]
  RewriteRule ^/(patternmatch.*) balancer://backend/$1 [P,L]
  RewriteRule ^/(.*) http://10.50.4.10/$1 [P,L]

If one of the nodes in balancer://backend dies, then typically it does
not fail over fast enough to stop the proxy snowballing, and when it
does failover, typically the second node then snowballs & dies from
the sudden influx of banked up requests.

I did some tests setting smax and max for the BalancerMembers - I set
smax=5 max=10, and then used ab to hit it with 50 concurrent hits -
both the Proxy and the back end showed 50 active connections. So
either smax/max aren't taking effect, or I'm completely
misinterpreting what they are supposed to do.

We're using Apache 2.2.3 under Debian 4.0 (Etch) - and apparently the
Prefork version.

Any ideas/help would be greatly appreciated. Thanks!

-- 
Trevor Phillips  - http://dortamur.livejournal.com/
"On nights such as this, evil deeds are done. And good deeds, of
course. But mostly evil, on the whole."
      -- (Terry Pratchett, Wyrd Sisters)

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org