httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colm MacCárthaigh <colmm...@Redbrick.DCU.IE>
Subject Re: Linux + TCP_CORK + IPv6 = Broken [PATCH]
Date Wed, 04 Dec 2002 12:25:49 GMT
On Wed, Dec 04, 2002 at 07:12:04AM -0500, Jeff Trawick wrote:
> Colm MacCárthaigh <colmmacc@Redbrick.DCU.IE> writes:
> 
> > O.k., now that is interesting, another thing I've noticed is that when
> > the encoding is chunked, I can't replicate the problem. Presumably the
> > problem is related to how much data is trying to be sent down the
> > socket at once.
> 
> Well, note that Apache uses different socket APIs for dynamic content
> vs. static content, and chunking is only used for dynamic content.
> That should be related to the breakage.
> 
> Also, the other poster mentioned that SSL works over IPv6 but non-SSL
> doesn't on one box.  That also corresponds to the same difference in
> socket API usage.
> 
> We can't use sendfile for any dynamic content.  We can't use sendfile
> for encrypted static or encrypted dynamic content.  So it really seems
> that sendfile is key to the issue.

Looks very much like that is the case.

> But I'm confused because you already disabled sendfile usage and that
> didn't bring some amount of happiness.  Also, the path where we use
> sendfile is the path where we use TCP_CORK so how can there be a
> problem if we don't use sendfile()?
> 
> double-checking where TCP_CORK is used:
> 
> 1) APR_TCP_NOPUSH_FLAG is TCP_CORK on Linux
> 2) APR_TCP_NOPUSH socket option controls APR_TCP_NOPUSH_FLAG
> 3) apr_sendfile() for Linux is the only code that dorks with
>    APR_TCP_NOPUSH (and thus TCP_CORK)

Here's what I see , when I use --disable-sendfile, this is a successful
request:

poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1
read(8, "GET / HTTP/1.1\r\n", 8000)     = 16
gettimeofday({1039003895, 580353}, NULL) = 0
poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1
read(8, "Host: www.heanet.ie\r\n", 8000) = 21
poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1
read(8, "\r\n", 8000)                   = 2
stat64("/usr/local/apache/htdocs/", {st_mode=S_IFDIR|S_ISGID|0755,
st_size=4096, ...}) = 0
brk(0x814c000)                          = 0x814c000
stat64("/usr/local/apache/htdocs/index.html", {st_mode=S_IFREG|0644,
st_size=256, ...}) = 0
open("/usr/local/apache/htdocs/index.html", O_RDONLY) = 9
brk(0x814f000)                          = 0x814f000
read(8, 0x8146a30, 8000)                = -1 EAGAIN (Resource
temporarily unavailable)
setsockopt(8, SOL_TCP, TCP_NODELAY, [0], 4) = 0
setsockopt(8, SOL_TCP, TCP_CORK, [1], 4) = 0
writev(8, [{"HTTP/1.1 200 OK\r\nDate: Wed, 04 D"..., 248}], 1) = 248
sendfile(8, 9, [0], 256)                = 256
setsockopt(8, SOL_TCP, TCP_CORK, [0], 4) = 0
setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0
write(4, "2001:770:18:1:201:2ff:fef0:f281 "..., 90) = 90
poll(^C
 <unfinished ...>

and this is an unsuccessful one:

poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1
read(8, "GET / HTTP/1.1\r\n", 8000)     = 16
gettimeofday({1039003973, 629255}, NULL) = 0
poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1
read(8, "Host: www.heanet.ie\r\n", 8000) = 21
poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 300000) = 1
read(8, "\r\n", 8000)                   = 2
stat64("/usr/local/apache/htdocs/", {st_mode=S_IFDIR|S_ISGID|0755,
st_size=4096, ...}) = 0
brk(0x814c000)                          = 0x814c000
stat64("/usr/local/apache/htdocs/index.html", {st_mode=S_IFREG|0644,
st_size=256, ...}) = 0
open("/usr/local/apache/htdocs/index.html", O_RDONLY) = 9
brk(0x814f000)                          = 0x814f000
read(8, 0x8146a30, 8000)                = -1 EAGAIN (Resource
temporarily unavailable)
setsockopt(8, SOL_TCP, TCP_NODELAY, [0], 4) = 0
setsockopt(8, SOL_TCP, TCP_CORK, [1], 4) = 0
writev(8, [{"HTTP/1.1 200 OK\r\nDate: Wed, 04 D"..., 248}], 1) = 248
sendfile(8, 9, [0], 256)                = 256
setsockopt(8, SOL_TCP, TCP_CORK, [0], 4) = 0
setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0
write(4, "2001:770:18:1:201:2ff:fef0:f281 "..., 90) = 90
poll([{fd=8, events=POLLIN, revents=POLLIN}], 1, 15000) = 1
read(8, "", 8000)                       = 0
gettimeofday({1039003977, 946871}, NULL) = 0
shutdown(8, 1 /* send */)               = 0
poll([{fd=8, events=POLLIN, revents=POLLIN|POLLHUP}], 1, 2000) = 1
read(8, "", 512)                        = 0
close(8)                                = 0
read(5, 0xbffffb77, 1)                  = -1 EAGAIN (Resource
temporarily unavailable)
close(9)                                = 0
accept(3,  <unfinished ...>

> So I still don't understand why disabling sendfile usage didn't avoid
> the problem, unless it isn't specific to TCP_CORK after all.

Looks like the --disable-sendfile doesnt work, it's still using
sendfile. Reading the configure.in tells me you meant 
--without-sendfile ;-) Which does do as would be expected, the
problem goes away. 

I've written a patch, to confirm that it is really
TCP_CORK, applied it, and tested it, and when I use the new
--disable-ipv6-tcp-cork option, everything is nice and happy :-)
Wether I use sendfile or not.

My tests (and patch) were based on apr and apr-util from CVS , 
with the 2.0.43 codebase, because CVS seems broken right now.

Index: configure.in
===================================================================
RCS file: /home/cvspublic/apr/configure.in,v
retrieving revision 1.506
diff -u -r1.506 configure.in
--- configure.in	2 Dec 2002 16:07:09 -0000	1.506
+++ configure.in	4 Dec 2002 10:29:45 -0000
@@ -1770,6 +1770,16 @@
 echo "${nl}Checking for IPv6 Networking support..."
 dnl Start of checking for IPv6 support...
 
+ipv6_tcp_cork=1
+AC_ARG_ENABLE(ipv6-tcp-cork, 
+  [  --disable-ipv6-tcp-cork Disable TCP_CORK with IPv6.],
+  [  if test "$enableval" = "no"; then
+        use_ipv6_tcp_cork=0
+     fi ],
+  [ use_ipv6_tcp_cork=1 ] )
+
+AC_SUBST(use_ipv6_tcp_cork)
+
 AC_ARG_ENABLE(ipv6,
   [  --disable-ipv6          Disable IPv6 support in APR.],
   [ if test "$enableval" = "no"; then
Index: include/apr.h.in
===================================================================
RCS file: /home/cvspublic/apr/include/apr.h.in,v
retrieving revision 1.117
diff -u -r1.117 apr.h.in
--- include/apr.h.in	22 Oct 2002 12:37:40 -0000	1.117
+++ include/apr.h.in	4 Dec 2002 10:29:46 -0000
@@ -171,6 +171,11 @@
  */
 #define APR_TCP_NOPUSH_FLAG       @apr_tcp_nopush_flag@
 
+/* Should we use "corked" TCP with IPv6 ? (this seems to be broken on
+ * linux
+ */
+#define APR_USE_IPV6_TCP_CORK     @use_ipv6_tcp_cork@
+
 /* Is the TCP_NODELAY socket option inherited from listening sockets?
 */
 #define APR_TCP_NODELAY_INHERITED @tcp_nodelay_inherited@
Index: network_io/unix/sockopt.c
===================================================================
RCS file: /home/cvspublic/apr/network_io/unix/sockopt.c,v
retrieving revision 1.63
diff -u -r1.63 sockopt.c
--- network_io/unix/sockopt.c	20 Nov 2002 03:50:21 -0000	1.63
+++ network_io/unix/sockopt.c	4 Dec 2002 10:29:46 -0000
@@ -259,9 +259,15 @@
         return APR_ENOTIMPL;
 #endif
     }
-    if (opt & APR_TCP_NOPUSH) {
+    if (opt & APR_TCP_NOPUSH
+#ifndef APR_USE_IPV6_TCP_CORK
+		    sd;fl sd;klfjss ds;lkfjds
+        && sock->remoteaddr->sa.sin.sin_family != APR_INET6
+#endif
+		    ) {
+    
 #if APR_TCP_NOPUSH_FLAG
-        if (apr_is_option_set(sock->netmask, APR_TCP_NOPUSH) != on) {
+       	if (apr_is_option_set(sock->netmask, APR_TCP_NOPUSH) != on) {
             int optlevel = IPPROTO_TCP;
             int optname = TCP_NODELAY;
-- 
colmmacc@redbrick.dcu.ie        PubKey: colmmacc+pgp@redbrick.dcu.ie  
Web:                                 http://devnull.redbrick.dcu.ie/ 

Mime
View raw message