Mailing-List: contact dev-help@subversion.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: neutral (athena.apache.org: local policy)
From: "Bert Huijben" <bert@qqmail.nl>
To: "'Ivan Zhakov'" <ivan@visualsvn.com>,
	"'Johan Corveleyn'" <jcorvel@gmail.com>
Cc: "'Subversion Development'" <dev@subversion.apache.org>
References: 
 <CAB84uBV6dfnLg8f=NsMN84u-XcSmyUAvBuvNSbg3x9O9P90aeA@mail.gmail.com>
 <004401cd83e8$2d6fb7c0$884f2740$@qqmail.nl>
 <CAB84uBWiKQ3+ZX04b3K4V2YSCYvXkqEbCoCjCa0C6tmHdtO_wQ@mail.gmail.com>
 <CAB84uBWS0sMaDWb6n00dP2i=RzrLhMr2RbGLb+t_4Dav0JOqeg@mail.gmail.com>
 <CABw-3YdKU49QZComX=3guARVqcXO1-RcWoaLS+Y-kZ-W-08ykA@mail.gmail.com>
In-Reply-To: 
 <CABw-3YdKU49QZComX=3guARVqcXO1-RcWoaLS+Y-kZ-W-08ykA@mail.gmail.com>
Subject: RE: notification output bottleneck
Date: Wed, 29 Aug 2012 00:58:32 +0200
Message-ID: <02ff01cd8570$a7bf5ce0$f73e16a0$@qqmail.nl>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Thread-Index: AQGpUYibH63GL/ePTdZaUXq2g3lShQKpe2BIAKMEgeICC3cH1gDDoz+5l4de6fA=
Content-Language: nl


> -----Original Message-----
> From: Ivan Zhakov [mailto:ivan@visualsvn.com]
> Sent: woensdag 29 augustus 2012 00:47
> To: Johan Corveleyn
> Cc: Bert Huijben; Subversion Development
> Subject: Re: notification output bottleneck
>=20
> On Wed, Aug 29, 2012 at 2:00 AM, Johan Corveleyn <jcorvel@gmail.com>
> wrote:
> > On Mon, Aug 27, 2012 at 10:45 AM, Johan Corveleyn =
<jcorvel@gmail.com>
> wrote:
> >> On Mon, Aug 27, 2012 at 2:09 AM, Bert Huijben <bert@qqmail.nl> =
wrote:
> >>>> -----Original Message-----
> >>>> From: Johan Corveleyn [mailto:jcorvel@gmail.com]
> >>>> Sent: maandag 27 augustus 2012 00:39
> >>>> To: Subversion Development
> >>>> Subject: notification output bottleneck
> >>>>
> >>>> I've never noticed this before (slow server, but now I'm testing =
a
> >>>> new,
> >>> faster
> >>>> server), but it seems that the writing of notification output on
> >>>> stdout is
> >>> a
> >>>> bottleneck for checkout, update or export on Windows (cmd.exe).
> >>>> With a fast server, hot caches, and everything on a Gb LAN,
> >>>> checkout of a large working copy is twice as fast (from 79 =
minutes
> >>>> down to 35 minutes) when I redirect output to NUL. I tested with
> >>>> both 1.7.5 (SlikSVN) and trunk, the results were about the same.
> >>>>
> >>>> Is there anything that could be done in svn to remove that
> >>>> bottleneck (buffering, ...)?
> >>>>
> >>>> (there might have been some discussion about this before, but I
> >>>> can't find
> >>> it)
> >>>
> >>> I found this problem too while profiling.
> >>>
> >>> The console apis on Windows in the Microsoft libraries are slow
> >>> because they try to be safe for multithreaded applications. There
> >>> are some defines that can be used to speed this up (which will =
avoid
> >>> the most costly thread synchronisations), but we can't just apply
> >>> these to our code as that would make libsvn_subr no longer =
multithread
> safe.
> >>
> >> Interesting.
> >>
> >> Am I correct in saying that this is really only a problem for the
> >> commandline 'svn' client (and perhaps 'svnadmin' and 'svnlook' etc
> >> ...)? Other clients can do their own fast notification handling,
> >> right?
> >>
> >> And Isn't 'svn' always single-threaded?
> >>
> >> In that case: couldn't we make the necessary multithread-unsafe, =
but
> >> fast, output functions for 'svn' only?
> >
> > Is the above idea possible / sane / desirable?
> >
> > Bert, what are those defines to speed up console access at the =
expense
> > of thread safety? I'd like to give them a quick try just to see how
> > much it would impact.
> >
> > ISTR from before we migrated to SVN that CVS did not have this
> > problem: console output would just fly by very quickly. At least
> > that's how I remember it :-), I haven't run CVS for some time now
> > (it's also possible that I am misremembering, or that it is because =
I
> > was running the client on WinXP back then).
> >
> One difference that Subversion uses direct unbuffered WriteFile to =
write to
> stdout instead of buffered printf()/puts(). You may try to tweak
> svn_stream_for_stdout() to return buffered stream and check =
performance
> difference.
>=20
> Other possible reason that Subversion converts all texts from utf8 to =
console
> encoding before writing text to stdout.

The time isn't spend in Subversion, but in the thread synchronisation in =
the CRT library.

I'm not sure when you tested CVS, but mostly everything became much =
faster over the last few years, while the console subsystem didn't and =
relatively slowed down a bit. Thread synchronization is much cheaper =
when you have a single non-hyperthreaded CPU, compared to the current =
processors.

Microsoft didn't really bother optimizing this as for most Windows =
applications console IO is never a bottleneck. Outputting to a console =
much faster than a user can read is fast enough. 10 times faster than =
unreadable fast has no additional profit.

Visual C++ 6.0 did have a single threaded C++ library as a link time =
option. This library was removed some 8 or 10 years ago. After that they =
have the defines as a replacement for some key scenarios. (I don't know =
a pointer to look for them in the documentation, but they are documented =
in the IO library on MSDN. You can also check the defines in the header =
files.)

	Bert
>=20
> --
> Ivan Zhakov