Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 156DC200D08 for ; Thu, 21 Sep 2017 23:06:19 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 13CB01609E1; Thu, 21 Sep 2017 21:06:19 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0A6041609DB for ; Thu, 21 Sep 2017 23:06:17 +0200 (CEST) Received: (qmail 242 invoked by uid 500); 21 Sep 2017 21:06:17 -0000 Mailing-List: contact users-help@subversion.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list users@subversion.apache.org Received: (qmail 231 invoked by uid 99); 21 Sep 2017 21:06:16 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Sep 2017 21:06:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 1F95018FB94 for ; Thu, 21 Sep 2017 21:06:16 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.379 X-Spam-Level: ** X-Spam-Status: No, score=2.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id jZSUsjX_xQHn for ; Thu, 21 Sep 2017 21:06:14 +0000 (UTC) Received: from mail-ua0-f179.google.com (mail-ua0-f179.google.com [209.85.217.179]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 3BCA65FBBB for ; Thu, 21 Sep 2017 21:06:14 +0000 (UTC) Received: by mail-ua0-f179.google.com with SMTP id q29so4389180uaf.3 for ; Thu, 21 Sep 2017 14:06:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=lg18/jBGcTS7AHRVnOKuib24HvrkYeLzf+Tyis0DC6w=; b=Ewl8Xh1bjV6nYJmy/LePgmBLztLbwtY96nQftUiDVcbO+MNK34ttNcuUAFvgBTWKeW y6rebA7gVfRNFQsNpl+3fv8zIFQ1Bi/3/jbnhaMZnWJ4jzaFDlSWemsWDF4zCrPFHITI LmRijHMAKLFliGvu38vctg/XF0djpT2cToADRRyuXf4UWXLjFjwkZj2zoerSm/4Vk503 62L6R50jU2q1FqBcBuF7v9FjWF3mGvJGYbx+ZD7Zq6H7HsFpOiU4/vmMZxlzp8gwYuOw ARE3feJvBjIJkZ1LZMD1Gkgx1pq2FQWZ1UyDTThYDZsbt7fDOiVkxKsPkJxeU9mMcYuX ymAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=lg18/jBGcTS7AHRVnOKuib24HvrkYeLzf+Tyis0DC6w=; b=peD8nNoUoQT1zX3KoMuC05Q3nwZILTN84GK4cAkwLQs6Y5nQ1XIWyC7tV39vNT8HIn xrvHo5FJT/4yixQppyToymxtcKCL/m+9t/c3oVeBBZZkETTIoHZi9EOly6U3Sntvjiq9 2zFFfk+G57Lgobg1gXhIdYDgyVFHCJpY6sMCmiCWWuobP7MG5hnENKRnuhi56sJa5lSj 4K1OL/2jaQdVCLEUuoirKo6ZPl9aMjVUIxnXW24zvuedIaOSQcw3RNQabAafT0qPSx4J 5g24lyyVzL+9T8XDmuDjprTHe2nfutNJLcyBjSX+GHieSfFfdcZxYB2N9O5sabR5ltPQ Qy+w== X-Gm-Message-State: AHPjjUgm0eFXJVIxCj+gkYwSepG/xBOhc4e20SB9WwMnivGQw+2T2NEE gDkYLOF/Xu4DJwDD3cgIPKjl17v6EEMElnI4SWM= X-Google-Smtp-Source: AOwi7QAk5CT26UISa8HqyTjGbXOgZeQjs9kAyFya/gjnzw4n85PuO1Mgv4a2Q54ni59RKVgxIbI2wm8cMVN4xs8pYY4= X-Received: by 10.176.75.202 with SMTP id b10mr3324145uag.174.1506027973529; Thu, 21 Sep 2017 14:06:13 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.80.140 with HTTP; Thu, 21 Sep 2017 14:06:13 -0700 (PDT) Received: by 10.103.80.140 with HTTP; Thu, 21 Sep 2017 14:06:13 -0700 (PDT) In-Reply-To: <6EC193E9-ECDE-4FB5-99B5-C4FEFD128321@ietcombustion.com> References: <1B1E129E-7BA7-438D-B38F-90C0E4A47933@ietcombustion.com> <6EC193E9-ECDE-4FB5-99B5-C4FEFD128321@ietcombustion.com> From: Johan Corveleyn Date: Thu, 21 Sep 2017 23:06:13 +0200 Message-ID: Subject: Fwd: Re: Subversion svn+ssh, sshd 100% CPU To: users@subversion.apache.org Content-Type: multipart/alternative; boundary="f403045e1f7a6254fe0559b9760e" archived-at: Thu, 21 Sep 2017 21:06:19 -0000 --f403045e1f7a6254fe0559b9760e Content-Type: text/plain; charset="UTF-8" Forwarding to the list (Zoran, please use reply all to keep the list in cc). ---------- Doorgestuurd bericht ---------- Van: "Zoran Petkovic" Datum: 21 sep. 2017 19:01 Onderwerp: Re: Subversion svn+ssh, sshd 100% CPU Aan: "Johan Corveleyn" Cc: Correction, I meant to say KBytes/s. -- Zoran Petkovic Industrial Engineering Technology Pty Ltd +61 413 254 315 | zoran@ietcombustion.com On 21/09/2017, 4:17 PM, "Zoran Petkovic" wrote: My results show that Linux clients stress the server less and complete the checkout quicker (5-10% on sshd and svnserve). Windows clients connecting remotely start off fast 2000-3000kbit/s then slow down to maximum 800kbit/s, the sshd process running at 100%, while the svnserve at 5%. So, I believe it is an issue with ssh rather than svn. Skelta mode seems to refer to apache, however these clients connect over ssh. -- Zoran Petkovic Industrial Engineering Technology Pty Ltd +61 413 254 315 | zoran@ietcombustion.com On 14/09/2017, 10:10 PM, "Johan Corveleyn" wrote: On Wed, Sep 13, 2017 at 7:20 AM, Matt Simmons wrote: > On Tue, Sep 12, 2017 at 12:14 PM Zoran Petkovic < zoran@ietcombustion.com> > wrote: >> >> In the past few days I have been doing extensive testing of Subversion >> with different clients, operating systems, client and server versions and >> have noticed very strange behaviour with windows clients connecting to Linux >> servers, hitting them with excessive CPU usage on the sshd process, where >> the Linux clients do not exhibit this behaviour. >> >> >> >> A sample test setup is as follows: >> >> Server Linux Ubuntu 16.04.3 LTS, OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL >> 1.0.2g 1 Mar 2016, Subversion version 1.9.3 (and 1.9.7). >> >> Client TortoiseSVN 1.9.7 >> >> >> >> When checking out large repositories the linux server is hit on the sshd >> process, the process running with 100% cpu usage. This in effect slows down >> the performance and ultimately the speed at which the checkout runs. Linux >> clients connecting to the same server do not cause this load on the server. >> >> >> >> This happens even when compressions is turned off and when encryption >> Cyphers are changed, as well as different versions of subversion. The >> behaviour is identical. I'm not sure who to address for this issue as this >> not only happens with TortoiseSVN but with SlikSVN as well. Any direction >> would be appreciated. > > Why does iostat show? Could it be that your underlying disk is io-saturated > and your CPU spike is due to iowait? One possible explanation would be that the windows clients perform the checkout faster than the linux clients for some reason, thus being able to put higher load on the server, saturating its cpu or disk I/O. Other than that I have no idea. The Linux client you're testing is also 1.9.7? Perhaps the windows client and the linux client are performing the checkout in a different way for some reason (I'm thinking of the so-called "skelta mode" vs. "bulk mode", see http://subversion.apache.org/docs/release-notes/1.8.html# serf-skelta-default). -- Johan --f403045e1f7a6254fe0559b9760e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Forwarding to the list (Zoran, please use reply all = to keep the list in cc).


---------- Doorgestuurd berich= t ----------
Van: "Zoran Petkovic" <zoran@ietcombustion.com>
Datum: 21 sep. 2017 = 19:01
Onderwerp: Re: Subversion svn+ssh, sshd 100% CPU
Aan: "Joh= an Corveleyn" <jcorvel@gmail.c= om>
Cc:

Cor= rection, I meant to say KBytes/s.

--
Zoran Petkovic
Industrial Engineering Technology Pty Ltd
+61 413 254 = 315=C2=A0<tel:0413%20254%20315> | zoran@ietcombustion.com


On 21/09/2017, 4:17 PM, "Zoran Petkov= ic" <zoran@ietcombustion= .com> wrote:

=C2=A0 =C2=A0 My results show that Linux clients stress the server less and= complete the checkout quicker (5-10% on sshd and svnserve). Windows client= s connecting remotely start off fast 2000-3000kbit/s then slow down to maxi= mum 800kbit/s, the sshd process running at 100%, while the svnserve at 5%. = So, I believe it is an issue with ssh rather than svn. Skelta mode seems to= refer to apache, however these clients connect over ssh.

=C2=A0 =C2=A0 --
=C2=A0 =C2=A0 Zoran Petkovic
=C2=A0 =C2=A0 Industrial Engineering Technology Pty Ltd
=C2=A0 =C2=A0 +61 413 254 315 <tel:0413%20254%20315> | zoran@ietcombustion.com


=C2=A0 =C2=A0 On 14/09/2017, 10:10 PM, "Johan Corveleyn" <jcorvel@gmail.com> wrote:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 On Wed, Sep 13, 2017 at 7:20 AM, Matt Simmons &= lt;bandman@gmail.com> wrote: =C2=A0 =C2=A0 =C2=A0 =C2=A0 > On Tue, Sep 12, 2017 at 12:14 PM Zoran Pet= kovic <zoran@ietcombustion.co= m>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 > wrote:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> In the past few days I have been doing= extensive testing of Subversion
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> with different clients, operating syst= ems, client and server versions and
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> have noticed very strange behaviour wi= th windows clients connecting to Linux
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> servers, hitting them with excessive C= PU usage on the sshd process, where
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> the Linux clients do not exhibit this = behaviour.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> A sample test setup is as follows:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> Server Linux Ubuntu 16.04.3 LTS, OpenS= SH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> 1.0.2g=C2=A0 1 Mar 2016, Subversion ve= rsion 1.9.3 (and 1.9.7).
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> Client TortoiseSVN 1.9.7
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> When checking out large repositories t= he linux server is hit on the sshd
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> process, the process running with 100%= cpu usage. This in effect slows down
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> the performance and ultimately the spe= ed at which the checkout runs. Linux
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> clients connecting to the same server = do not cause this load on the server.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> This happens even when compressions is= turned off and when encryption
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> Cyphers are changed, as well as differ= ent versions of subversion. The
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> behaviour is identical. I'm not su= re who to address for this issue as this
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> not only happens with TortoiseSVN but = with SlikSVN as well. Any direction
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >> would be appreciated.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >
=C2=A0 =C2=A0 =C2=A0 =C2=A0 > Why does iostat show? Could it be that you= r underlying disk is io-saturated
=C2=A0 =C2=A0 =C2=A0 =C2=A0 > and your CPU spike is due to iowait?

=C2=A0 =C2=A0 =C2=A0 =C2=A0 One possible explanation would be that the wind= ows clients perform the
=C2=A0 =C2=A0 =C2=A0 =C2=A0 checkout faster than the linux clients for some= reason, thus being
=C2=A0 =C2=A0 =C2=A0 =C2=A0 able to put higher load on the server, saturati= ng its cpu or disk I/O.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Other than that I have no idea.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 The Linux client you're testing is also 1.9= .7?

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Perhaps the windows client and the linux client= are performing the
=C2=A0 =C2=A0 =C2=A0 =C2=A0 checkout in a different way for some reason (I&= #39;m thinking of the
=C2=A0 =C2=A0 =C2=A0 =C2=A0 so-called "skelta mode" vs. "bul= k mode", see
=C2=A0 =C2=A0 =C2=A0 =C2=A0 http://subversion.apache.org/docs/release-notes/1.8.html#serf= -skelta-default).

=C2=A0 =C2=A0 =C2=A0 =C2=A0 --
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Johan





--f403045e1f7a6254fe0559b9760e--