httpd-bugs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject DO NOT REPLY [Bug 49417] New: mod_proxy crosses connections when apache times out
Date Wed, 09 Jun 2010 15:26:14 GMT
https://issues.apache.org/bugzilla/show_bug.cgi?id=49417

           Summary: mod_proxy crosses connections when apache times out
           Product: Apache httpd-2
           Version: 2.2.15
          Platform: PC
        OS/Version: Windows Server 2003
            Status: NEW
          Severity: critical
          Priority: P2
         Component: mod_proxy
        AssignedTo: bugs@httpd.apache.org
        ReportedBy: loren.anderson@mscibarra.com


As part of our software security remediations, we upgraded our site to use
apache 2.2.13 from 2.2.2.

We found a critical issue with Apache 2.2.13 (and reproduced on 2.2.15). This
issue does not exist in apache 2.2.2.

The basics of this problem are that when you use mod_proxy to connect to a
backend tomcat server and apache is configured to timeout before tomcat, a
connection to the backend tomcat server is placed back into the pool of
existing connections with what ever data that was destined to the original
caller waiting to be retrieved by the next caller to take this connection from
the pool. In our application which is statefull, this lead to customer's
sessions crossing and customer A saw customer B's data.

Setting DisableReuse=true or changing the timeout of apache to be larger that
the tomcat server avoids this issue.

I was able to recreate the issue with a simple TOMCAT echo servlet that takes
an incoming URL which contains a random number and returning that value in the
body of an HTML response. The apache and tomcat servers run on a single Windows
7 box (I have also tried this on Windows Server 2003) configured to connect via
localhost.

The client driver needs to have KEEPALIVE on (since the front end connection
and back end connections seem to be linked from a life-span perspective). My
sample client driver was written in ruby. It generates a random number and then
checks that the response it gets is what it send. 

Apache is configured to have 5 pooled connections (max=5, min=5) and set to
timeout in 10 seconds. TOMCAT is configured to timeout in 5 minutes. 

The client ruby code sleeps a random amount of time between 1 and 15 seconds.
This leads to about 1/3 of the clients timeing out. I run 20 clients. Within a
few seconds, the problem is reproduced (client does not see the number it send
it).

Here is my apache configuration:

=== BEGIN httpd.conf ===
PidFile httpd.pid

Timeout 600
KeepAlive On
MaxKeepAliveRequests 0
KeepAliveTimeout 300
ServerTokens ProductOnly

FileETag -INode MTime Size

ServerSignature Off

ThreadsPerChild 1000
MaxRequestsPerChild  0

Listen 81

LoadModule authz_host_module    "modules/mod_authz_host.so"
LoadModule proxy_module         "modules/mod_proxy.so"
LoadModule proxy_http_module    "modules/mod_proxy_http.so"
LoadModule dir_module           "modules/mod_dir.so"
LoadModule log_config_module    "modules/mod_log_config.so"
LoadModule mime_module          "modules/mod_mime.so"
LoadModule rewrite_module       "modules/mod_rewrite.so"
LoadModule headers_module       "modules/mod_headers.so"

RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

TypesConfig "conf/mime.types"

AddDefaultCharset   ISO-8859-1
MaxMemFree 1000000

ErrorLog "apache-error.log"
LogLevel info
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
combined
LogFormat "%h %A %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog "apache-access-BK-PEWEB01-B1_APACHE.log" common
DocumentRoot "root"

<Directory "root">
    Options FollowSymLinks Includes
    AllowOverride None
    Order allow,deny
    Allow from all
    DirectoryIndex index.html
</Directory>

<Proxy *>
     AddDefaultCharset Off
     Order deny,allow
     Allow from all
</Proxy>

ProxyPass /test http://localhost:9090/test min=5 max=5 timeout=10
ProxyPassReverse /test http://localhost:9090/test
=== END httpd.conf ===

Here is the TOMCAT configuration:

=== BEGIN server.xml ===
<Server
  port="8005"
  shutdown="SHUTDOWN"
  debug="0"
>
    <Service name="Tomcat">
        <Connector
            port="9090"
            maxThreads="1000"
            minSpareThreads="25"
            maxSpareThreads="75"
            enableLookups="false"
            redirectPort="8443"
            acceptCount="100"
            debug="0"
            connectionTimeout="300000"
            disableUploadTimeout="true"
            server="THX1138"
            proxyName="pe1.barra.com"
            proxyPort="443"
        />
        <Engine
          name="Standalone"
          defaultHost="localhost"
          debug="0"
        >
            <Host
              name="localhost"
              debug="0"
              appBase="webapps"
              unpackWARs="true"
            >
                <Context
                  useHttpOnly="true"
                  reloadable="false"
                  path="/test"
                  docBase="c:/workspace/webapps/test"
                  debug="0"
                  privileged="true"
                />
            </Host>
        </Engine>
    </Service>
</Server>
=== END server.xml ===

Here is my simple eacho servlet:

=== BEGIN EchoServlet.java ===
import com.sun.mail.util.UUEncoderStream;
import org.apache.derby.iapi.util.StringUtil;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Random;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class EchoServlet extends javax.servlet.http.HttpServlet implements
javax.servlet.Servlet {
    private static Random rand = new Random(System.nanoTime());
    public EchoServlet() {
        super();
    }

    protected void doGet(HttpServletRequest request, HttpServletResponse
response) throws ServletException, IOException {
        doPost(request, response);
    }

    protected void doPost(HttpServletRequest request, HttpServletResponse
response) throws ServletException, IOException {
        response.setContentType("text/html");

        String uri = request.getRequestURI();

        if(uri.startsWith("/test/")) {
            String num = uri.replace("/test/", "");
            long rid = Long.parseLong(num);
            PrintWriter writer = response.getWriter();
            try {
                Thread.sleep(1000 * rand.nextInt(17));
            } catch(InterruptedException e) {}
            writer.print("<html>");
            writer.print("<body>");
            writer.print(rid);
            writer.print("</body>");
            writer.print("</html>");
            writer.close();
        }
    }
}
=== END EchoServlet.java ===

Here is my client ruby driver:

=== BEGIN echoclient.rb ===
require "socket"
require "cgi"

$printHeader = false
class Response
  attr_reader :socket, :header, :content

  def initialize socket
    @socket = socket
    @header = @content = ""
    contentlength = 0

    while (line = @socket.gets) != nil
      break if line == "\r\n"
      puts line if $printHeader
      @header << line
      case line
      when /^content-length: (\d+)/i
        contentlength = $1.to_i
      end
    end
    if contentlength != 0
      @content = @socket.read contentlength
    end
  end
end

socket = nil
connected = false

while true
    if socket == nil || socket.closed?
      connected = false
      socket = nil
      print '='
      $stdout.flush
    end
    while !connected
       begin
            socket = TCPSocket.new('localhost', 81)
            connected = true
        rescue
            puts "Got error, sleeping"
            sleep 5
            next
        end
    end
    num = rand(2000000000)
    #puts "Input: #{num}"
    req = "GET /test/#{num} HTTP/1.1\r\nConnection: Keep-Alive\r\nHost:
127.0.0.1\r\n\r\n"
    socket.write req
    socket.flush

    resp_num = 0
    response = Response.new socket
    if response.content.length == 0
      print '&'
      $stdout.flush
    end
    if response.content =~ /<HTML><BODY>([0-9]+)<\/BODY><\/HTML>/io
            resp_num = $1.to_i
    end
    if resp_num == 0
      $printHeader = true
      print '0'
    elsif num != resp_num
      $printHeader = false
      puts num
      puts resp_num
      print '-'
    else
      $printHeader = false
      print '*'
    end
    $stdout.flush
end
=== END echoclient.rb ===

-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


Mime
View raw message