httpd-bugs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject [Bug 61616] New: mod_proxy_connect: stall and connection loss on bi-directional traffic
Date Sat, 14 Oct 2017 10:09:36 GMT
https://bz.apache.org/bugzilla/show_bug.cgi?id=61616

            Bug ID: 61616
           Summary: mod_proxy_connect: stall and connection loss on
                    bi-directional traffic
           Product: Apache httpd-2
           Version: 2.4.25
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: mod_proxy_connect
          Assignee: bugs@httpd.apache.org
          Reporter: carsten@wolffcarsten.de
  Target Milestone: ---

Created attachment 35421
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=35421&action=edit
default vhost with proxy_connect conf

When proxy_connect is used for taffic that is less of a request-response
nature, it can happen, that the send buffer on the backend socket gets full. In
that case, in libapr, when the writev() comes back with EAGAIN, it will block
in poll() on the backend socket with a 300-second timeout. No read from any of
the two sockets can happen anymore in that situation. Thus, depending on the
application protocol that is tunneled through CONNECT, the backend application
will then fillup the receive buffer at proxy_connect's backend socket and then
stall trying to send indefinitely. Now proxy_connect and the backend
application are in a deadlock.

A concrete setup to reproduce it (using debian stretch):

rsync -------------------------> apache ---------------> rsync --daemon
        CONNECT localhost:873

I will attach configuration and a script to this report and can also provide a
qemu image, if desired.

In this setup the stall looks like this:

----------------------------- rsync ----------------------------------------

# RSYNC_PROXY=localhost:80 rsync -rcP rsync://localhost:873/testsrc/ /testdst/
[..]
5269
              0   0%    0.00kB/s    0:00:00  
[ .. 5 minute delay .. ]
rsync: connection unexpectedly closed (403374822 bytes received so far)
[receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(235)
[receiver=3.1.2]
rsync: connection unexpectedly closed (242925 bytes received so far)
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(235)
[generator=3.1.2]

----------------------------- netstat --------------------------------------

# netstat -pnt | grep -v :22
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
PID/Program name    

tcp6       0         0 ::1:43642               ::1:80                 
ESTABLISHED 21638/rsync
tcp6  169814         0 ::1:80                  ::1:43642              
ESTABLISHED 20628/apache2
tcp6  5974823  2608576 ::1:43798               ::1:873                
ESTABLISHED 20628/apache2
tcp6  809935   2760413 ::1:873                 ::1:43798              
ESTABLISHED 21639/rsync

----------------------------- strace --------------------------------------

[..]
read(10,
"\24\372\342\1\16\255\223\375G\213k\251\215\360\357}\311\275\202\5jQ>\341>\370
jr\357,\4"..., 8000) = 8000    <---- fd 10 is client socket
writev(11, [{iov_base= [..] ., iov_len=7424}], 16) = -1 EAGAIN (Resource
temporarily unavailable)                      <---- fd 11 is backend socket
poll([{fd=11, events=POLLOUT}], 1, 300000)
 <detached ...>

----------------------------- gdb -----------------------------------------

(gdb) bt
#0  0x00007f37a88c4690 in __poll_nocancel () at
../sysdeps/unix/syscall-template.S:84
#1  0x00007f37a8dc5088 in apr_poll () from
target:/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2  0x00005572f5f781cd in send_brigade_blocking (c=0x7f37a95ad9b0,
bytes_written=0x7f37a95ade68, bb=0x7f37a95a1068, s=0x7f37a95a6c28) at
core_filters.c:747
#3  ap_core_output_filter (f=0x7f37a95add88, new_bb=0x7f37a95a1068) at
core_filters.c:542
#4  0x00007f37a58fac77 in ap_proxy_transfer_between_connections
(r=r@entry=0x7f37a95a50a0, c_i=0x7f37a95ad290, c_o=0x7f37a95ad9b0,
bb_i=<optimized out>, bb_o=0x7f37a95a1068, name=0x7f37a56e39dd "client",
sent=0x0, 
    bsize=8192, after=1) at proxy_util.c:3836
#5  0x00007f37a56e31c3 in proxy_connect_handler (r=0x7f37a95a50a0,
worker=<optimized out>, conf=<optimized out>, url=<optimized out>,
proxyname=<optimized out>, proxyport=<optimized out>) at
mod_proxy_connect.c:421

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


Mime
View raw message