httpd-bugs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject DO NOT REPLY [Bug 47167] New: Authenticated sessions being switched by reverse proxy
Date Fri, 08 May 2009 04:56:18 GMT
https://issues.apache.org/bugzilla/show_bug.cgi?id=47167

           Summary: Authenticated sessions being switched by reverse proxy
           Product: Apache httpd-2
           Version: 2.2.11
          Platform: PC
        OS/Version: Windows XP
            Status: NEW
          Severity: normal
          Priority: P2
         Component: mod_proxy
        AssignedTo: bugs@httpd.apache.org
        ReportedBy: duncan.fletcher@dsto.defence.gov.au


We've got a Subversion server (mod_dav_svn under Apache on Windows Server 2003)
set up behind another Apache server configured as a reverse proxy (also on
Windows Server 2003).  We recently upgraded the reverse-proxy box from Apache
httpd 2.0.54 to httpd 2.2.11 and found that when two users access the
Subversion repository at the same time, their usernames (or session tags
maybe?) are being switched internally by Apache.  i.e. if the two users are
simultaneously each trying to access an area that the other doesn't have access
to, they get back 401 Authorization Required errors, with the wrong usernames
appearing in the log (further explanation below).  The Subversion server has
been running Apache 2.2.11 for months without trouble, its just the
reverse-proxy we've changed.  We've isolated this error down to one of either
mod_proxy or mod_proxy_http (details below), so even though the problem comes
up when we use Subversion (including a custom authorization module), we're
pretty sure the bug lies in Apache.

At the moment we're fine sticking with Apache 2.0.63 (which happens to be a lot
faster than 2.2.6 as a reverse proxy, about 2x speed difference), but if anyone
can look into this issue that'd be good.  Thanks!


Example scenario of problem/symptoms:
- userA is doing a large (500KB, 30+ files) commit to
http://svnserver/repos/sandboxes/userA/private/
- userB is doing a large (500KB, 30+ files) commit to
http://svnserver/repos/sandboxes/userB/private/
- one user kicks off their commit while the other commit is already running,
i.e. in the middle of setting up the transaction & transferring the files - or
both are kicked off at the same time - doesn't matter really, as long as both
sessions are in progress simultaneously
- At that point, one OR both users will get a 401 error (there is no particular
pattern we've seen, other than that the user who kicks off second is more
likely to get the 401 while the other user's transaction runs to completion,
but about 1/3 of the time they both die)
- Looking at the access.log of the subversion server, it shows that (for
example) userB was denied access to userA's sandbox
(http://svnserver/repos/sandboxes/userA/private/fileXYZ), even though userB was
actually operating on their own sandbox at the time.
- Snippet of access.log pasted in at the bottom. We can reproduce this fairly
easily if additional Apache logging is necessary, but unfortunately we can't
supply any network-traffic dumps.


Details:
- Size of commit/transaction doesn't actually matter, its just that longer
transactions are more likely to collide in testing.
- Problem only occurs if accessing the svn repository via the reverse proxy.
i.e. Accessing the repository directly has no problems.
- Provided the commit is large enough that there is proper overlap, this
happens every time, its not intermittent. The only thing that changes between
tests is how many files the two commits get through before dying, but it always
dies in less than a second.
- No related messages in error.log (just the usual startup and shutdown
notices).
- Tested against different versions of Apache httpd and found some interesting
results:
    Apache 2.0.54    - okay
    Apache 2.0.63    - okay
    Apache 2.2.2    - problem
    Apache 2.2.3    - problem
    Apache 2.2.4    - problem
    Apache 2.2.6    - okay!!
    Apache 2.2.8    - problem
    Apache 2.2.9    - problem
    Apache 2.2.10    - problem
    Apache 2.2.11    - problem
    Apache 2.2.11 with mod_proxy.so and mod_proxy_http.so copied in from Apache
2.2.6 - okay!!
- i.e.the problem is in either mod_proxy.so or mod_proxy_http.so and affects
every release of Apache 2.2 except for 2.2.6!
- Authentication of users is via mod_auth_sspi, i.e. Windows Domain
authentication, at the Subversion server.
- Apache installation is from the binaries downloaded from
http://apache.16degrees.com.au/httpd/binaries/win32/ and
http://archive.apache.org/dist/httpd/binaries/win32/.
- Tried with and without the openssl build, no difference.
- For testing, both the subversion server and the reverse-proxy server were put
on the same LAN subnet so that test-clients could access either/both (to
confirm the problem only happened when the reverse proxy was in the loop)
- Just for info completeness, we're running svn 1.5.4 but have also checked svn
1.6.1: no difference in the problem.
- We'd been running the reverse-proxy with Apache 2.0.54 for over a year with
no trouble until the upgrade to 2.2.11; using the same httpd.conf as below
(except for the version-specific changes like ServerRoot and mod_access vs
mod_authz_host).
- Reverse-proxy to the svnserver is the only job the problematic Apache
installation is doing (I've listed below all of the modules it has loaded).

Relevant conf lines from the reverse-proxy server:
--vv------<snip>---------vv--
LoadModule authz_host_module modules/mod_authz_host.so LoadModule dir_module
modules/mod_dir.so LoadModule env_module modules/mod_env.so LoadModule
headers_module modules/mod_headers.so LoadModule include_module
modules/mod_include.so LoadModule log_config_module modules/mod_log_config.so
LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module
modules/mod_proxy_http.so LoadModule proxy_html_module
modules/mod_proxy_html.so LoadModule setenvif_module modules/mod_setenvif.so
ProxyRequests off ProxyVia On <Location /repos >
    ProxyPass http://svnserver.ourdomain/repos
      ProxyPassReverse /repos/
      ProxyPassReverse http://svnserver/repos
      ProxyPassReverse http://svnserver.ourdomain/repos
      <Limit OPTIONS PROPFIND GET REPORT MKACTIVITY PROPPATCH PUT CHECKOUT
MKCOL MOVE COPY DELETE LOCK UNLOCK MERGE>
        Order Deny,Allow
        Allow from all
        Satisfy Any
      </Limit>
</Location>
--^^------<snip>---------^^--


Relevant conf lines from subversion server ("svnserver") (note these are not
the only modules this server has loaded):
--vv------<snip>---------vv--
LoadModule sspi_auth_module modules/mod_auth_sspi.so 
LoadModule dav_svn_module modules/mod_dav_svn.so 
LoadModule permissions_svn_module modules/mod_permissions_svn.so 
<Location /repos>
    DAV svn
    SVNPath //fileserver/svnrepos/repos
    AuthType SSPI
    AuthName "Subversion repository by OurDomain"
    Require valid-user
    SSPIAuth On            
    SSPIAuthoritative On    
    SSPIDomain OurDomain
    SSPIOfferBasic On        
    PermissionsSVNAccessFile //fileserver/svnrepos/conf/svs_permissions.conf
</Location>
--^^------<snip>---------^^--


mod_permissions_svn.so is an in-house simple-ish authorization module based on
mod_authz_svn that hooks into httpd core as follows:
--vv------<snip>---------vv--
static void registerHooks(apr_pool_t *p)
{
    ap_hook_access_checker(checkAccess, NULL, NULL, APR_HOOK_LAST);
    ap_hook_auth_checker(checkAuthorisation, NULL, NULL, APR_HOOK_FIRST); 
}

module AP_MODULE_DECLARE_DATA permissions_svn_module = {
    STANDARD20_MODULE_STUFF,
    createConfigRecord,              /* config creater per dir */
    NULL,                            /* dir merger --- default is to override
*/
    NULL,                            /* server config */
    NULL,                            /* merge server config */
    MODULE_ARGUMENTS,                /* command apr_table_t */
    registerHooks                    /* register hooks */
};
--^^------<snip>---------^^--
where checkAccess(...) and checkAuthorisation(...) implement our custom rules
using fields from the request_rec parameter such as user, uri, etc.  I only
mention this to help convince anyone reading this that this mod_permissions_svn
is not the problem.


And finally a snippet of the access.log of the subversion server...
--vv------<snip>---------vv--
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "PROPFIND
/repos/sandboxes/fletched/private/tool/test/test_compatibility/TestPlugin.cpp
HTTP/1.1" 207 776
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "PROPFIND
/repos/!svn/vcc/default HTTP/1.1" 207 455
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "PROPFIND
/repos/!svn/bc/52575/sandboxes/fletched/private/tool/test/test_compatibility/TestPlugin.cpp
HTTP/1.1" 207 535
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "CHECKOUT
/repos/!svn/ver/52575/sandboxes/fletched/private/tool/test/test_compatibility/TestPlugin.cpp
HTTP/1.1" 201 307
xxx.xxx.xxx.198 - OURDOMAIN\\coopern [27/Apr/2009:14:36:02 +1000] "PUT
/repos/!svn/wrk/11125f68-8c91-644c-85a7-b9a056d3b561/sandboxes/coopern/private/testing_commit/tool/test/test_csvfile.cpp
HTTP/1.1" 204 -
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "PUT
/repos/!svn/wrk/11125f68-8c91-644c-85a7-b9a056d3b561/sandboxes/coopern/private/testing_commit/tool/tool_marshal_containers.h
HTTP/1.1" 401 401
xxx.xxx.xxx.198 - OURDOMAIN\\coopern [27/Apr/2009:14:36:02 +1000] "PROPFIND
/repos/sandboxes/fletched/private/tool/test/test_compatibility/TestPlugin.h
HTTP/1.1" 401 401
xxx.xxx.xxx.198 - OURDOMAIN\\coopern [27/Apr/2009:14:36:02 +1000] "DELETE
/repos/!svn/act/10089f74-f41f-484e-b1a0-783dafcf2155 HTTP/1.1" 204 -
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "DELETE
/repos/!svn/act/11125f68-8c91-644c-85a7-b9a056d3b561 HTTP/1.1" 204 -
--^^------<snip>---------^^--

-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


Mime
View raw message