Return-Path: Delivered-To: apmail-httpd-dev-archive@httpd.apache.org Received: (qmail 78437 invoked by uid 500); 4 Dec 2002 12:25:54 -0000 Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: dev@httpd.apache.org list-help: list-unsubscribe: list-post: Delivered-To: mailing list dev@httpd.apache.org Received: (qmail 78424 invoked from network); 4 Dec 2002 12:25:54 -0000 Date: Wed, 4 Dec 2002 12:25:49 +0000 From: =?iso-8859-1?Q?Colm_MacC=E1rthaigh?= To: dev@httpd.apache.org Cc: trawick@attglobal.net, dev@apr.apache.org Subject: Re: Linux + TCP_CORK + IPv6 = Broken [PATCH] Message-ID: <20021204122549.A28623@prodigy.Redbrick.DCU.IE> References: <20021203213808.GA2974@byron.heanet.ie> <004d01c29b1c$002046a0$210d640a@unfix.org> <20021204092734.A17772@prodigy.Redbrick.DCU.IE> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.2.5i In-Reply-To: ; from trawick@attglobal.net on Wed, Dec 04, 2002 at 07:12:04AM -0500 X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N On Wed, Dec 04, 2002 at 07:12:04AM -0500, Jeff Trawick wrote: > Colm MacC�rthaigh writes: > > > O.k., now that is interesting, another thing I've noticed is that when > > the encoding is chunked, I can't replicate the problem. Presumably the > > problem is related to how much data is trying to be sent down the > > socket at once. > > Well, note that Apache uses different socket APIs for dynamic content > vs. static content, and chunking is only used for dynamic content. > That should be related to the breakage. > > Also, the other poster mentioned that SSL works over IPv6 but non-SSL > doesn't on one box. That also corresponds to the same difference in > socket API usage. > > We can't use sendfile for any dynamic content. We can't use sendfile > for encrypted static or encrypted dynamic content. So it really seems > that sendfile is key to the issue. Looks very much like that is the case. > But I'm confused because you already disabled sendfile usage and that > didn't bring some amount of happiness. Also, the path where we use > sendfile is the path where we use TCP_CORK so how can there be a > problem if we don't use sendfile()? > > double-checking where TCP_CORK is used: > > 1) APR_TCP_NOPUSH_FLAG is TCP_CORK on Linux > 2) APR_TCP_NOPUSH socket option controls APR_TCP_NOPUSH_FLAG > 3) apr_sendfile() for Linux is the only code that dorks with > APR_TCP_NOPUSH (and thus TCP_CORK) Here's what I see , when I use --disable-sendfile, this is a successful request: poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1 read(8, "GET / HTTP/1.1\r\n", 8000) = 16 gettimeofday({1039003895, 580353}, NULL) = 0 poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1 read(8, "Host: www.heanet.ie\r\n", 8000) = 21 poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1 read(8, "\r\n", 8000) = 2 stat64("/usr/local/apache/htdocs/", {st_mode=S_IFDIR|S_ISGID|0755, st_size=4096, ...}) = 0 brk(0x814c000) = 0x814c000 stat64("/usr/local/apache/htdocs/index.html", {st_mode=S_IFREG|0644, st_size=256, ...}) = 0 open("/usr/local/apache/htdocs/index.html", O_RDONLY) = 9 brk(0x814f000) = 0x814f000 read(8, 0x8146a30, 8000) = -1 EAGAIN (Resource temporarily unavailable) setsockopt(8, SOL_TCP, TCP_NODELAY, [0], 4) = 0 setsockopt(8, SOL_TCP, TCP_CORK, [1], 4) = 0 writev(8, [{"HTTP/1.1 200 OK\r\nDate: Wed, 04 D"..., 248}], 1) = 248 sendfile(8, 9, [0], 256) = 256 setsockopt(8, SOL_TCP, TCP_CORK, [0], 4) = 0 setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 write(4, "2001:770:18:1:201:2ff:fef0:f281 "..., 90) = 90 poll(^C and this is an unsuccessful one: poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1 read(8, "GET / HTTP/1.1\r\n", 8000) = 16 gettimeofday({1039003973, 629255}, NULL) = 0 poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1 read(8, "Host: www.heanet.ie\r\n", 8000) = 21 poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1 read(8, "\r\n", 8000) = 2 stat64("/usr/local/apache/htdocs/", {st_mode=S_IFDIR|S_ISGID|0755, st_size=4096, ...}) = 0 brk(0x814c000) = 0x814c000 stat64("/usr/local/apache/htdocs/index.html", {st_mode=S_IFREG|0644, st_size=256, ...}) = 0 open("/usr/local/apache/htdocs/index.html", O_RDONLY) = 9 brk(0x814f000) = 0x814f000 read(8, 0x8146a30, 8000) = -1 EAGAIN (Resource temporarily unavailable) setsockopt(8, SOL_TCP, TCP_NODELAY, [0], 4) = 0 setsockopt(8, SOL_TCP, TCP_CORK, [1], 4) = 0 writev(8, [{"HTTP/1.1 200 OK\r\nDate: Wed, 04 D"..., 248}], 1) = 248 sendfile(8, 9, [0], 256) = 256 setsockopt(8, SOL_TCP, TCP_CORK, [0], 4) = 0 setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0 write(4, "2001:770:18:1:201:2ff:fef0:f281 "..., 90) = 90 poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 15000) = 1 read(8, "", 8000) = 0 gettimeofday({1039003977, 946871}, NULL) = 0 shutdown(8, 1 /* send */) = 0 poll([{fd=8, events=POLLIN, revents=POLLIN|POLLHUP}], 1, 2000) = 1 read(8, "", 512) = 0 close(8) = 0 read(5, 0xbffffb77, 1) = -1 EAGAIN (Resource temporarily unavailable) close(9) = 0 accept(3, > So I still don't understand why disabling sendfile usage didn't avoid > the problem, unless it isn't specific to TCP_CORK after all. Looks like the --disable-sendfile doesnt work, it's still using sendfile. Reading the configure.in tells me you meant --without-sendfile ;-) Which does do as would be expected, the problem goes away. I've written a patch, to confirm that it is really TCP_CORK, applied it, and tested it, and when I use the new --disable-ipv6-tcp-cork option, everything is nice and happy :-) Wether I use sendfile or not. My tests (and patch) were based on apr and apr-util from CVS , with the 2.0.43 codebase, because CVS seems broken right now. Index: configure.in =================================================================== RCS file: /home/cvspublic/apr/configure.in,v retrieving revision 1.506 diff -u -r1.506 configure.in --- configure.in 2 Dec 2002 16:07:09 -0000 1.506 +++ configure.in 4 Dec 2002 10:29:45 -0000 @@ -1770,6 +1770,16 @@ echo "${nl}Checking for IPv6 Networking support..." dnl Start of checking for IPv6 support... +ipv6_tcp_cork=1 +AC_ARG_ENABLE(ipv6-tcp-cork, + [ --disable-ipv6-tcp-cork Disable TCP_CORK with IPv6.], + [ if test "$enableval" = "no"; then + use_ipv6_tcp_cork=0 + fi ], + [ use_ipv6_tcp_cork=1 ] ) + +AC_SUBST(use_ipv6_tcp_cork) + AC_ARG_ENABLE(ipv6, [ --disable-ipv6 Disable IPv6 support in APR.], [ if test "$enableval" = "no"; then Index: include/apr.h.in =================================================================== RCS file: /home/cvspublic/apr/include/apr.h.in,v retrieving revision 1.117 diff -u -r1.117 apr.h.in --- include/apr.h.in 22 Oct 2002 12:37:40 -0000 1.117 +++ include/apr.h.in 4 Dec 2002 10:29:46 -0000 @@ -171,6 +171,11 @@ */ #define APR_TCP_NOPUSH_FLAG @apr_tcp_nopush_flag@ +/* Should we use "corked" TCP with IPv6 ? (this seems to be broken on + * linux + */ +#define APR_USE_IPV6_TCP_CORK @use_ipv6_tcp_cork@ + /* Is the TCP_NODELAY socket option inherited from listening sockets? */ #define APR_TCP_NODELAY_INHERITED @tcp_nodelay_inherited@ Index: network_io/unix/sockopt.c =================================================================== RCS file: /home/cvspublic/apr/network_io/unix/sockopt.c,v retrieving revision 1.63 diff -u -r1.63 sockopt.c --- network_io/unix/sockopt.c 20 Nov 2002 03:50:21 -0000 1.63 +++ network_io/unix/sockopt.c 4 Dec 2002 10:29:46 -0000 @@ -259,9 +259,15 @@ return APR_ENOTIMPL; #endif } - if (opt & APR_TCP_NOPUSH) { + if (opt & APR_TCP_NOPUSH +#ifndef APR_USE_IPV6_TCP_CORK + sd;fl sd;klfjss ds;lkfjds + && sock->remoteaddr->sa.sin.sin_family != APR_INET6 +#endif + ) { + #if APR_TCP_NOPUSH_FLAG - if (apr_is_option_set(sock->netmask, APR_TCP_NOPUSH) != on) { + if (apr_is_option_set(sock->netmask, APR_TCP_NOPUSH) != on) { int optlevel = IPPROTO_TCP; int optname = TCP_NODELAY; -- colmmacc@redbrick.dcu.ie PubKey: colmmacc+pgp@redbrick.dcu.ie Web: http://devnull.redbrick.dcu.ie/