From users-return-25228-apmail-subversion-users-archive=subversion.apache.org@subversion.apache.org Tue Oct 4 20:37:57 2016 Return-Path: X-Original-To: apmail-subversion-users-archive@minotaur.apache.org Delivered-To: apmail-subversion-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E698E1985D for ; Tue, 4 Oct 2016 20:37:56 +0000 (UTC) Received: (qmail 59518 invoked by uid 500); 4 Oct 2016 20:37:51 -0000 Delivered-To: apmail-subversion-users-archive@subversion.apache.org Received: (qmail 59479 invoked by uid 500); 4 Oct 2016 20:37:51 -0000 Mailing-List: contact users-help@subversion.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list users@subversion.apache.org Delivered-To: moderator for users@subversion.apache.org Received: (qmail 60411 invoked by uid 99); 4 Oct 2016 17:26:17 -0000 X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_NONE=-0.0001, RP_MATCHES_RCVD=-2.999] autolearn=disabled X-IronPort-AV: E=Sophos;i="5.31,444,1473120000"; d="scan'208,217";a="776098" From: Karen Pease To: "users@subversion.apache.org" Subject: svn+ssh very slow, but ssh is just fine - difference in socket calls? Thread-Topic: svn+ssh very slow, but ssh is just fine - difference in socket calls? Thread-Index: AQHSHmIUgX7vJcGNakylphrPMbIvpw== Date: Tue, 4 Oct 2016 17:26:04 +0000 Message-ID: <1475601964612.67180@isavia.is> Accept-Language: is-IS, en-US Content-Language: is-IS X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [192.168.106.165] Content-Type: multipart/alternative; boundary="_000_147560196461267180isaviais_" MIME-Version: 1.0 --_000_147560196461267180isaviais_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable As it says on the tin, our connections via svn+ssh are painfully slow, yet = we can ssh into the server without any delays whatsoever. A find on the su= bversion repository likewise whips through without delay, and there's no me= mory or CPU load on the server when people are connecting. So all signs po= int to some sort of connection or authentication delay. I ran ssh through strace and compared that to a svn connection through stra= ce. Here's where things go awry. First, SSH which works: SSH: socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) =3D 3 fcntl(3, F_SETFD, FD_CLOEXEC) =3D 0 connect(3, {sa_family=3DAF_INET, sin_port=3Dhtons(22), sin_addr=3Dinet_addr= ("{{Our IP}}")}, 16) =3D 0 setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) =3D 0 getpeername(3, {sa_family=3DAF_INET, sin_port=3Dhtons(22), sin_addr=3Dinet_= addr("{{Our IP}}")}, [128->16]) =3D 0 getpeername(3, {sa_family=3DAF_INET, sin_port=3Dhtons(22), sin_addr=3Dinet_= addr("{{Our IP}}")}, [128->16]) =3D 0 getpeername(3, {sa_family=3DAF_INET, sin_port=3Dhtons(22), sin_addr=3Dinet_= addr("{{Our IP}}")}, [128->16]) =3D 0 stat("/home/fdps/.ssh", {st_mode=3DS_IFDIR|0700, st_size=3D4096, ...}) =3D = 0 open("/etc/passwd", O_RDONLY|O_CLOEXEC) =3D 4 ... and so forth; it keeps on going normally. But with svn+ssh: socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 3 connect(3, {sa_family=3DAF_INET, sin_port=3Dhtons(53), sin_addr=3Dinet_addr= ("{{Our IP}}")}, 16) =3D 0 poll([{fd=3D3, events=3DPOLLOUT}], 1, 0) =3D 1 ([{fd=3D3, revents=3DPOLL= OUT}]) sendmmsg(3, [{msg_hdr=3D{msg_name=3DNULL, msg_namelen=3D0, msg_iov=3D[{iov_= base=3D"\24T\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\1\0\1", iov_len=3D26}], msg= _iovlen=3D1, msg_controllen=3D0, msg_flags=3DMSG_CTRUNC|MSG_DONTWAIT|MSG_EO= R|MSG_FIN|MSG_RST|MSG_ERRQUEUE|MSG_MORE|MSG_CMSG_CLOEXEC|0x13ba0010}, msg_l= en=3D26}, {msg_hdr=3D{msg_name=3DNULL, msg_namelen=3D0, msg_iov=3D[{iov_bas= e=3D"\211\205\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\34\0\1", iov_len=3D26}], m= sg_iovlen=3D1, msg_controllen=3D0, msg_flags=3DMSG_OOB|MSG_PEEK|MSG_DONTROU= TE|MSG_EOR|MSG_RST|MSG_ERRQUEUE|MSG_NOSIGNAL|MSG_MORE|MSG_BATCH|MSG_FASTOPE= N|MSG_CMSG_CLOEXEC|0x1c920000}, msg_len=3D26}], 2, MSG_NOSIGNAL) =3D 2 poll([{fd=3D3, events=3DPOLLIN}], 1, 5000) =3D 0 (Timeout) socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 6 connect(6, {sa_family=3DAF_INET, sin_port=3Dhtons(53), sin_addr=3Dinet_addr= ("{{Our IP}}")}, 16) =3D 0 poll([{fd=3D6, events=3DPOLLOUT}], 1, 0) =3D 1 ([{fd=3D6, revents=3DPOLL= OUT}]) sendmmsg(6, [{msg_hdr=3D{msg_name=3DNULL, msg_namelen=3D0, msg_iov=3D[{iov_= base=3D"\24T\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\1\0\1", iov_len=3D26}], msg= _iovlen=3D1, msg_controllen=3D0, msg_flags=3DMSG_CTRUNC|MSG_DONTWAIT|MSG_EO= R|MSG_FIN|MSG_RST|MSG_ERRQUEUE|MSG_MORE|MSG_CMSG_CLOEXEC|0x13ba0010}, msg_l= en=3D26}, {msg_hdr=3D{msg_name=3DNULL, msg_namelen=3D0, msg_iov=3D[{iov_bas= e=3D"\211\205\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\34\0\1", iov_len=3D26}], m= sg_iovlen=3D1, msg_controllen=3D0, msg_flags=3DMSG_OOB|MSG_PEEK|MSG_DONTROU= TE|MSG_EOR|MSG_RST|MSG_ERRQUEUE|MSG_NOSIGNAL|MSG_MORE|MSG_BATCH|MSG_FASTOPE= N|MSG_CMSG_CLOEXEC|0x1c920000}, msg_len=3D26}], 2, MSG_NOSIGNAL) =3D 2 ... and so on, stuck in repeated timeouts until it gives up. Now, note the difference on the socket call. Ssh uses a socket call with S= OCK_STREAM / TCP, while svn+ssh uses SOCK_DGRAM|SOCK_NONBLOCK / IP. Why wo= uld it do that? That doesn't make any sense to me. How can I change this? For a list of all socket() calls from each: SSH: socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3 socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 3 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) =3D 3 socket(AF_UNIX, SOCK_STREAM, 0) =3D 4 socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 4 socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 4 socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 4 socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 4 SVN+SSH: socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3 socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 3 socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 3 socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 6 socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC|SOCK_NONBLOCK, NETLINK_ROUTE) =3D = 3 socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) =3D 3 socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) =3D 3 socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) =3D 3 socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) =3D 3 socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) =3D 3 - kv, Karen --_000_147560196461267180isaviais_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

As it says on the tin, our connections via svn+ssh are painfully slo= w, yet we can ssh into the server without any delays whatsoever.  A fi= nd on the subversion repository likewise whips through without delay, and t= here's no memory or CPU load on the server when people are connecting.  So all signs point to some sort of conne= ction or authentication delay.


I ran ssh through strace and compared that to a svn connection through s= trace.  Here's where things go awry.  First, SSH which works:


SSH:


socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) =3D 3
fcntl(3, F_SETFD, FD_CLOEXEC)       &nbs= p;   =3D 0
connect(3, {sa_family=3DAF_INET, sin_port=3Dhtons(22), sin_addr=3Dinet_addr= ("{{Our IP}}")}, 16) =3D 0
setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) =3D 0
getpeername(3, {sa_family=3DAF_INET, sin_port=3Dhtons(22), sin_addr=3Dinet_= addr("{{Our IP}}")}, [128->16]) =3D 0
getpeername(3, {sa_family=3DAF_INET, sin_port=3Dhtons(22), sin_addr=3Dinet_= addr("{{Our IP}}")}, [128->16]) =3D 0
getpeername(3, {sa_family=3DAF_INET, sin_port=3Dhtons(22), sin_addr=3Dinet_= addr("{{Our IP}}")}, [128->16]) =3D 0
stat("/home/fdps/.ssh", {st_mode=3DS_IFDIR|0700, st_size=3D4096, = ...}) =3D 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) =3D 4


... and so forth; it keeps on going normally.  But with svn+ssh= :


socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 3
connect(3, {sa_family=3DAF_INET, sin_port=3Dhtons(53), sin_addr=3Dinet_addr= ("{{Our IP}}")}, 16) =3D 0
poll([{fd=3D3, events=3DPOLLOUT}], 1, 0)    =3D 1 ([{fd=3D3,= revents=3DPOLLOUT}])
sendmmsg(3, [{msg_hdr=3D{msg_name=3DNULL, msg_namelen=3D0, msg_iov=3D[{iov_= base=3D"\24T\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\1\0\1", iov_len= =3D26}], msg_iovlen=3D1, msg_controllen=3D0, msg_flags=3DMSG_CTRUNC|MSG_DON= TWAIT|MSG_EOR|MSG_FIN|MSG_RST|MSG_ERRQUEUE|MSG_MORE|MSG_CMSG_CLOEXEC|0x13ba= 0010}, msg_len=3D26}, {msg_hdr=3D{msg_name=3DNULL, msg_namelen=3D0, msg_iov=3D[{i= ov_base=3D"\211\205\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\34\0\1", i= ov_len=3D26}], msg_iovlen=3D1, msg_controllen=3D0, msg_flags=3DMSG_OOB|MSG_= PEEK|MSG_DONTROUTE|MSG_EOR|MSG_RST|MSG_ERRQUEUE|MSG_NOSIGNAL|MSG_MORE|MSG_B= ATCH|MSG_FASTOPEN|MSG_CMSG_CLOEXEC|0x1c920000}, msg_len=3D26}], 2, MSG_NOSIGNAL) =3D 2
poll([{fd=3D3, events=3DPOLLIN}], 1, 5000)  =3D 0 (Timeout)
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 6
connect(6, {sa_family=3DAF_INET, sin_port=3Dhtons(53), sin_addr=3Dinet_addr= ("{{Our IP}}")}, 16) =3D 0
poll([{fd=3D6, events=3DPOLLOUT}], 1, 0)    =3D 1 ([{fd=3D6,= revents=3DPOLLOUT}])
sendmmsg(6, [{msg_hdr=3D{msg_name=3DNULL, msg_namelen=3D0, msg_iov=3D[{iov_= base=3D"\24T\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\1\0\1", iov_len= =3D26}], msg_iovlen=3D1, msg_controllen=3D0, msg_flags=3DMSG_CTRUNC|MSG_DON= TWAIT|MSG_EOR|MSG_FIN|MSG_RST|MSG_ERRQUEUE|MSG_MORE|MSG_CMSG_CLOEXEC|0x13ba= 0010}, msg_len=3D26}, {msg_hdr=3D{msg_name=3DNULL, msg_namelen=3D0, msg_iov=3D[{i= ov_base=3D"\211\205\1\0\0\1\0\0\0\0\0\0\10spathiwa\0\0\34\0\1", i= ov_len=3D26}], msg_iovlen=3D1, msg_controllen=3D0, msg_flags=3DMSG_OOB|MSG_= PEEK|MSG_DONTROUTE|MSG_EOR|MSG_RST|MSG_ERRQUEUE|MSG_NOSIGNAL|MSG_MORE|MSG_B= ATCH|MSG_FASTOPEN|MSG_CMSG_CLOEXEC|0x1c920000}, msg_len=3D26}], 2, MSG_NOSIGNAL) =3D 2


... and so on, stuck in repeated timeouts until it gives up.  =


Now, note the difference on the socket call.  Ssh uses a socket cal= l with SOCK_STREAM / TCP, while svn+ssh uses SOCK_DGRAM|SOCK_NONBLOCK /= IP.  Why would it do that?  That doesn't make any sense to me. &= nbsp;How can I change this?  


For a list of all socket() calls from each:


SSH:


socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 3
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) =3D 3
socket(AF_UNIX, SOCK_STREAM, 0)       &n= bsp; =3D 4
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 4
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 4
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 4
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 4


SVN+SSH:


socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) =3D 3
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 3
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 3
socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 6
socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC|SOCK_NONBLOCK, NETLINK_ROUTE) =3D = 3
socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) =3D 3
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) =3D 3
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) =3D 3
socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) =3D 3
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) =3D 3


 - kv, Karen



--_000_147560196461267180isaviais_--