Return-Path: X-Original-To: apmail-subversion-users-archive@minotaur.apache.org Delivered-To: apmail-subversion-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 432049333 for ; Sat, 1 Oct 2011 23:33:39 +0000 (UTC) Received: (qmail 7372 invoked by uid 500); 1 Oct 2011 23:33:38 -0000 Delivered-To: apmail-subversion-users-archive@subversion.apache.org Received: (qmail 7344 invoked by uid 500); 1 Oct 2011 23:33:38 -0000 Mailing-List: contact users-help@subversion.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list users@subversion.apache.org Received: (qmail 7337 invoked by uid 99); 1 Oct 2011 23:33:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Oct 2011 23:33:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kyle.leber@gmail.com designates 209.85.210.171 as permitted sender) Received: from [209.85.210.171] (HELO mail-iy0-f171.google.com) (209.85.210.171) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Oct 2011 23:33:31 +0000 Received: by iagv1 with SMTP id v1so5707637iag.16 for ; Sat, 01 Oct 2011 16:33:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=QmYbeKP1U4nK+bvZU6I1zt7JIBQ357RZQumXwgS5Q+Y=; b=EHJBlNS8vffa1xbpUn/Bs1DqBCLKrKnZwNfV2tDWBxwjTuNgzid3xRgi/hNI2iT0fH FToPZT4pnDODFgMl0V3brQp/EkZ5E7PHjlrqQavg9YMli+MGE/E6IU0J+4W1eQToqggr pD1rxRBvDudtZZytOAAYm8v0cjQBslBTk411A= MIME-Version: 1.0 Received: by 10.42.145.129 with SMTP id f1mr5130492icv.64.1317511990580; Sat, 01 Oct 2011 16:33:10 -0700 (PDT) Received: by 10.42.220.200 with HTTP; Sat, 1 Oct 2011 16:33:10 -0700 (PDT) In-Reply-To: <20111001191045.GA25864@daniel3.local> References: <20111001191045.GA25864@daniel3.local> Date: Sat, 1 Oct 2011 19:33:10 -0400 Message-ID: Subject: Re: svn merge operation extremely slow From: Kyle Leber To: Daniel Shahaf Cc: Johan Corveleyn , users@subversion.apache.org Content-Type: multipart/alternative; boundary=90e6ba613ace2f944104ae452a49 X-Virus-Checked: Checked by ClamAV on apache.org --90e6ba613ace2f944104ae452a49 Content-Type: text/plain; charset=ISO-8859-1 What method of profiling do you recommend? I have used gprof previously (it's been awhile) but am not familiar with the subversion project source code and build setup. Is the a online guide or wiki describing the preferred setup for performing this? Kyle On Sat, Oct 1, 2011 at 3:10 PM, Daniel Shahaf wrote: > Johan Corveleyn wrote on Sat, Oct 01, 2011 at 20:47:29 +0200: > > [ Please do not top-post on this list, i.e. please put your reply > > below or inline. More below ... ] > > > > On Sat, Oct 1, 2011 at 6:49 PM, Kyle Leber wrote: > > > On Fri, Sep 30, 2011 at 7:15 PM, Johan Corveleyn > wrote: > > >> > > >> On Fri, Sep 30, 2011 at 3:29 PM, Kyle Leber > wrote: > > >> > I've encountered what I think is a problem with subversion, but I'm > not > > >> > completely sure (and according to the online instructions I should > bring > > >> > it > > >> > up here prior to filing a bug). > > >> > > >> Actually, the instructions on > > >> http://subversion.apache.org/issue-tracker.html say that you should > > >> send your report to users@, not dev@. So I'm adding users@. Please > > >> drop dev@ from any further replies. > > >> > > >> > Basically, we're trying to merge a rather large collection of > > >> > fixes back in our trunk. I check out a fresh copy of the trunk, > > >> > then use the merge syntax: svn merge https://path/to/my/branch . > > >> > > > >> > This generally churns along just fine, but we occasionally get > > >> > hung up on medium sized binary files where the svn client jumps > > >> > to 100% cpu usage and sits on it for 3+ hours before moving on to > > >> > the next file. These files are anywhere from 3-10MB in size, so > > >> > not ridiculously huge. We generally have these files marked as > > >> > octet stream, but changing to text did not help the situation > > >> > when we tried that. > > >> > > > >> > I did find an old forum discussion about a potential issue that > > >> > could be related. I was wondering if this was ever addressed and > > >> > could it still be the same problem. Link is here: > > >> > http://www.svnforum.org/threads/36123-Slow-SVN-merge > > >> > > > >> > I'm using svn client 1.6.12. I looked at the online change log > > >> > up through the 1.7 alphas and didn't see any bug fixes that > > >> > sounded relevant. > > >> > > >> This could be a relevant change (listed in the 1.7 release notes, not > > >> in the change log): > > >> > > >> > http://subversion.apache.org/docs/release-notes/1.7.html#diff-optimizations > > >> > > >> Can you please try one of the 1.7 pre-release binaries, and see if it > > >> helps? See http://subversion.apache.org/packages.html#pre-release > > >> > > > Thanks, Johan. I tested with 1.7rc4 and it did not make any > perceptible > > > difference. Anything else I can try? > > > > Hm, that's unfortunate. > > > > Actually, it was to be expected that this wouldn't help, because the > > diff-optimizations in 1.7 only play a role when merging text files > > (and diffing and blaming). And you said those > > "files-that-make-merge-hang" are generally marked as octet-stream, and > > changing them to text made no difference. > > > > That seems to indicate that the 100% cpu usage on the client isn't > > spent in the diff code (unlike the forum thread that you linked to, > > where the poster tracked it down to libsvn_diff/lcs.c --- he would > > definitely have been helped by the 1.7 improvements). > > > > What does 'svn merge' do for binary files? I checked svn_wc__merge() > a few months ago and for binary files all it knew to do was > > (a) if mine == merge-left then set merged := merge-right > (b) invoke the configured diff3-cmd > (c) raise a conflict > > but it didn't do any line-based merge (per Johan's second response). > > > So there's another reason. Maybe it has something to do with (lots of) > > subtree mergeinfo? Can you verify if there is a lot of svn:mergeinfo > > on directories and files all over the place? > > > > Also: can you tell us what version is running on the server? > > > > Maybe other people on this list have had similar experiences, and can > > give some suggestions? > > > > Another line of thought: the algorithm for computing binary deltas > changed a few years ago, and I recall reading (on old bug reports?) > about some cases in which the delta combiner would be inefficient for > deltas generated by old servers --- i.e., it would be expensive to 'svn > cat' files that were committed to old servers in repositories that > haven't been dumped/loaded by a newer server. > > In any case: can you run the merge under a profiler and tell us in what > function(s) time is spent? > > Daniel > > > -- > > Johan > --90e6ba613ace2f944104ae452a49 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable What method of profiling do you recommend?=A0 I have used gprof previously = (it's been awhile) but am not familiar with the subversion project sour= ce code and build setup.=A0 Is the a online guide or wiki describing the pr= eferred setup for performing this?

Kyle

On Sat, Oct 1, 2011 at 3:10 PM, = Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
Johan Corveleyn wrote on Sat, Oct 01, 2011 at 20:47:29 +0200:
> [ Please do not top-post on this lis= t, i.e. please put your reply
> below or inline. More below ... ]
>
> On Sat, Oct 1, 2011 at 6:49 PM, Kyle Leber <kyle.leber@gmail.com> wrote:
> > On Fri, Sep 30, 2011 at 7:15 PM, Johan Corveleyn <jcorvel@gmail.com> wrote:
> >>
> >> On Fri, Sep 30, 2011 at 3:29 PM, Kyle Leber <kyle.leber@gmail.com> wrote:
> >> > I've encountered what I think is a problem with subv= ersion, but I'm not
> >> > completely sure (and according to the online instruction= s I should bring
> >> > it
> >> > up here prior to filing a bug).
> >>
> >> Actually, the instructions on
> >> http://subversion.apache.org/issue-tracker.html say th= at you should
> >> send your report to users@, not dev@. So I'm adding users= @. Please
> >> drop dev@ from any further replies.
> >>
> >> > Basically, we're trying to merge a rather large coll= ection of
> >> > fixes back in our trunk.=A0 I check out a fresh copy of = the trunk,
> >> > then use the merge syntax: svn merge https://path/to/my/branch .
> >> >
> >> > This generally churns along just fine, but we occasional= ly get
> >> > hung up on medium sized binary files where the svn clien= t jumps
> >> > to 100% cpu usage and sits on it for 3+ hours before mov= ing on to
> >> > the next file.=A0 These files are anywhere from 3-10MB i= n size, so
> >> > not ridiculously huge.=A0 We generally have these files = marked as
> >> > octet stream, but changing to text did not help the situ= ation
> >> > when we tried that.
> >> >
> >> > I did find an old forum discussion about a potential iss= ue that
> >> > could be related.=A0 I was wondering if this was ever ad= dressed and
> >> > could it still be the same problem.=A0 Link is here:
> >> > http://www.svnforum.org/threads/36123-Slow-SVN-m= erge
> >> >
> >> > I'm using svn client 1.6.12.=A0 I looked at the onli= ne change log
> >> > up through the 1.7 alphas and didn't see any bug fix= es that
> >> > sounded relevant.
> >>
> >> This could be a relevant change (listed in the 1.7 release no= tes, not
> >> in the change log):
> >>
> >> http://subversion.apache.org/d= ocs/release-notes/1.7.html#diff-optimizations
> >>
> >> Can you please try one of the 1.7 pre-release binaries, and s= ee if it
> >> helps? See http://subversion.apache.org/packages.ht= ml#pre-release
> >>
> > Thanks, Johan. =A0I tested with 1.7rc4 and it did not make any pe= rceptible
> > difference. =A0Anything else I can try?
>
> Hm, that's unfortunate.
>
> Actually, it was to be expected that this wouldn't help, because t= he
> diff-optimizations in 1.7 only play a role when merging text files
> (and diffing and blaming). And you said those
> "files-that-make-merge-hang" are generally marked as octet-s= tream, and
> changing them to text made no difference.
>
> That seems to indicate that the 100% cpu usage on the client isn't=
> spent in the diff code (unlike the forum thread that you linked to, > where the poster tracked it down to libsvn_diff/lcs.c --- he would
> definitely have been helped by the 1.7 improvements).
>

What does 'svn merge' do for binary files? =A0I checked= svn_wc__merge()
a few months ago and for binary files all it knew to do was

(a) if mine =3D=3D merge-left then set merged :=3D merge-right
(b) invoke the configured diff3-cmd
(c) raise a conflict

but it didn't do any line-based merge (per Johan's second response)= .

> So there's another reason. Maybe it has something to do with (lots= of)
> subtree mergeinfo? Can you verify if there is a lot of svn:mergeinfo > on directories and files all over the place?
>
> Also: can you tell us what version is running on the server?
>
> Maybe other people on this list have had similar experiences, and can<= br> > give some suggestions?
>

Another line of thought: the algorithm for computing binary deltas changed a few years ago, and I recall reading (on old bug reports?)
about some cases in which the delta combiner would be inefficient for
deltas generated by old servers --- i.e., it would be expensive to 'svn=
cat' files that were committed to old servers in repositories that
haven't been dumped/loaded by a newer server.

In any case: can you run the merge under a profiler and tell us in what
function(s) time is spent?

Daniel

> --
> Johan

--90e6ba613ace2f944104ae452a49--