httpd-bugs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject DO NOT REPLY [Bug 38227] - AJP proxy not thread-safe?
Date Thu, 13 Apr 2006 15:54:32 GMT
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38227>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=38227





------- Additional Comments From jim.robinson@stanford.edu  2006-04-13 15:54 -------
Folks,

I don't know if my previous followup on this bug was read or not, or
if people are already looking at it.  Without feedback, I can't tell
if I should keep pursuing this, follow up with more details, etc.

I can understand somebody telling me "No, we know this isn't the
problem" but I'm not hearing anything.  Without feedback I'm stuck not
knowing if this is a bug or a fatal misunderstanding on my part.

Given httpd 2.2.1 and the following settings:

  Server Version: Apache/2.2.1 (Unix) DAV/2 mod_ssl/2.2.1 OpenSSL/0.9.7a
  Server Built: Apr 6 2006 00:22:26
  Module Magic Number: 20051115:1
  Hostname/port: server.mydomain.org:80
  Timeouts: connection: 300    keep-alive: 300
  MPM Name: Worker
  MPM Information: Max Daemons: 2 Threaded: yes Forked: yes
  Server Architecture: 32-bit
  Server Root: /highwire/server/apache
  Config File: -c/-C directives
  Server Built With:
    -D APACHE_MPM_DIR="server/mpm/worker"
    -D APR_HAS_SENDFILE
    -D APR_HAS_MMAP
    -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
    -D APR_USE_SYSVSEM_SERIALIZE
    -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
    -D APR_HAS_OTHER_CHILD
    -D AP_HAVE_RELIABLE_PIPED_LOGS
    -D HTTPD_ROOT="/usr/local/apache/2.2.1"
    -D SUEXEC_BIN="/usr/local/apache/2.2.1/bin/suexec"
    -D DEFAULT_ERRORLOG="logs/error_log"
    -D AP_TYPES_CONFIG_FILE="conf/mime.types"
    -D SERVER_CONFIG_FILE="conf/httpd.conf"

It appears as though the mod_proxy pool of connections is not being
consistently reused when the balancer is used.  Given the
configuration:

  ServerLimit 2
  ThreadsPerChild 10

  <Proxy balancer://tomcat>
    BalancerMember ajp://server.mydomain.org:8009 route=server.mydomain.org
  </Proxy>

  <VirtualHost  vhost.mydomain.org:80>
    ServerName  vhost.mydomain.org

    RewriteEngine on

    RewriteCond %{ENV:HTTPD_BASE}/logs/myapp.pause -f
    RewriteRule .* - [forbidden,last]

    DocumentRoot vhosts/myapp/htdocs

    RewriteRule ^(/+test(?:/.*)?)$ /myapp$1 [env=internal:yes]

    RewriteCond %{ENV:internal} =yes
    RewriteRule ^(/+myapp/.*)  $1 [passthrough,last]
    RewriteRule ^/+myapp/(.*) /$1 [redirect=permanent,last]

    ProxyPass /myapp balancer://tomcat/myapp stickysession=JSESSIONID nofailover=on
    ProxyPassReverse / http://vhost.mydomain.org/myapp/
    ProxyPassReverseCookiePath /myapp/ /
  </VirtualHost>

My understanding of the documentation

  http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass

is that, with the above ServerLimit and ThreadPerChild limit, we
should never see more than 20 connections form between Apache and any
one BalancerMember target (2 servers x 10 threads = 20 total threads).

On startup, we see zero connections between Apache and Tomcat:

  ; netstat -t | tr -s " " | egrep "^tcp.*:8009 ESTABLISHED"
  ;

I make one request, and see one connection form:

  ; ab -n 1 http://vhost.mydomain.org/test
  ; netstat -t | tr -s " " | egrep "^tcp.*:8009 ESTABLISHED"
  tcp 0 0 server.mydomain.org:59165 server.mydomain.org:8009 ESTABLISHED

Now what happens if I make 20 concurrent requests? What I would expect to
see is up to 20 connections established, and no more.

  ; ab -c 20 -n 20 http://vhost.mydomain.org/test
  ; netstat -t | tr -s " " | egrep "^tcp.*:8009 ESTABLISHED"|wc -l
       21

If I make another 20 connections, I see the connections more than
double:

  ; ab -c 20 -n 20 http://vhost.mydomain.org/test
  ; netstat -t | tr -s " " | egrep "^tcp.*:8009 ESTABLISHED"|wc -l
       44

Now look what happens when I plug in min/smax/max settings into the
BalancerMember directive.  These are settings which I think the
documentation indicates are the defaults, given my configuration
above:

  <Proxy balancer://tomcat>
    BalancerMember ajp://server.mydomain.org:8009 route=server.mydomain.org
min=0 smax=10 max=10
  </Proxy>

After an apache restart, we have no connections, I make a bunch of
requests, and see 20 connections, and never more than that.  Even when
I make 50 concurrent requests, Apache simply queues and processes as
it should:

  ; netstat -t | tr -s " " | egrep "^tcp.*:8009 ESTABLISHED"|wc -l
        0

  ; ab -c 20 -n 20 http://vhost.mydomain.org/test
  ; netstat -t | tr -s " " | egrep "^tcp.*:8009 ESTABLISHED"|wc -l
       20

  ; ab -c 50 -n 50 http://vhost.mydomain.org/test
  ; netstat -t | tr -s " " | egrep "^tcp.*:8009 ESTABLISHED"|wc -l
       20

If we activate the balancer, and the balancer has only one Tomcat
target as in this case, shouldn't it should still be obeying the same
limits as defined in the configuration.  Otherwise we have a situation
where the number of proxy connections is unbounded, right?

Say Apache is configured to serve no more than 1024 requests at any
one moment in time, and we wanted to balance load between two Tomcat
servers.  What we'd probably do is set up two Tomcat servers which are
each configured to accept 1024 requests, but we would expect to see
apache balancer send ~512 requests to each Tomcat server.  What we
would *not* expect to see is Apache opening more connections to a
Tomcat server than Apache itself is configured to serve at any one
time.

In summary, it appears to be a bug to have to declare min/smax/max
values which are the same as the defaults per the documentation,
without which the number of connections opened appears to be
unbounded.

My examination of the code led me to believe that the source of the
problem is that the BalancerMember worker is not getting its hmax
value properly initialized, and that it is therefore bypassing the
pool of reusable connections when it is acquiring connections
(proxy_util.c:1758, in ap_proxy_acquire_connection).

If I add the patch I submitted in comment #14 and remove the
min/smax/max directives, I see the behavior I am expecting -- Apache
never opens more than 20 connections in all to its Tomcat server.

Looking at svn.apache.org, I see that initialization was actually
removed in a previous version, though the comments do not explain why
it is not appropriate to initialize the balancer worker:

 
http://svn.apache.org/viewcvs.cgi/httpd/httpd/trunk/modules/proxy/proxy_balancer.c?rev=105347&r1=105320&r2=105347

The only reason I can think of to have the current behavior is that
somehow the dynamic of balancing between multiple backend servers is
expected to be able to handle more connections -- but it just doesn't
make sense to me that it would ever be normal for Apache to open more
connections to any one backend server than Apache can serve itself.

If this isn't considered a bug, I'd very much appreciate it if someone
would point out the reason.

Thank you for your time,


Jim


-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


Mime
View raw message