Mailing-List: contact dev-help@subversion.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <4F91CC73.3060305@web.de>
Date: Fri, 20 Apr 2012 22:52:03 +0200
From: Stefan Fuhrmann <eqfox@web.de>
User-Agent: Mozilla/5.0 (Windows NT 6.0;
 rv:11.0) Gecko/20120327 Thunderbird/11.0.1
MIME-Version: 1.0
To: dev@subversion.apache.org
Subject: Re: Merge policies
References: <4F8FED8C.5080902@alice-dsl.de>
 <CAB84uBX2gx5ghPd-ZmMK6bL4D7+YtLAq7qWXLfO4CEiXugyxdg@mail.gmail.com>
In-Reply-To: 
 <CAB84uBX2gx5ghPd-ZmMK6bL4D7+YtLAq7qWXLfO4CEiXugyxdg@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Am 19.04.2012 14:03, schrieb Johan Corveleyn:
> On Thu, Apr 19, 2012 at 12:48 PM, Stefan Fuhrmann
> <stefanfuhrmann@alice-dsl.de>  wrote:
>> Hi all,
>>
>> After having a closer look at merge and discussing it
>> with Julian on IRC, there seems to be no silver bullet.
>> However, we identified a few things that could be changed
>> and set of constellations that make merge harder than
>> it needs to be.
>>
>> For the first, there will be another post soon. The second
>> boils down to policy. Luckily, SVN has a mechanism to
>> enforce policies: server-side hook scripts. My proposal
>> is to develop a small set of scripts that a user can
>> combine to prevent situations that her life harder than
>> necessary. This should give us enough time to improve
>> the merge logic inside the SVN libs.
>>
>> The following pre-commit scripts / policies would be useful.
>>
>> * Common parts [not a policy]
>>   We first check whether the commit contains a changed
>>   svn:merge-info property. This limits the performance
>>   impact on non-merge commits and we need to identify
>>   all changed svn:merge-info anyway.
>>
>>   Also, the merges that happened on the source branch
>>   from a different location than the target branch are
>>   of no interest for the policy checkers. E.g.:
>>
>>   r20: merge r19 from ^/sub-branch to ^/branch
>>   txn: merge r10-20 from ^/branch to ^/trunk
>>   Both merges will show up in the merge-info delta but
>>   we only need to evaluate the second one.
> I haven't read your mail in detail yet, but just a note in passing: in
> my experience (with a 1.5 server with FSFS-on-NFS backend) propgetting
> ('svnlook propget') every changed file during pre-commit can be very
> expensive. I ran into this with a script for enforcing our autoprops
> (like svn:eol-style=native on some files), with a combination of
> 'svnlook changed' and 'svnlook propget'. On our (admittedly not very
> fast) server, users started running into client-side timeouts because
> of this, when they committed 300 or so changed files (which happens
> easily when syncing branches). The default client-side timeout for
> neon is 30 seconds (which can be increased in the 'servers' file), and
> if the pre-commit hook takes longer than that, you're screwed.
>
> So I've optimized the pre-commit validation script to only 'propget'
> files from 'svnlook changed' which were either Added ('A'), or which
> had property modifications ('*U'). So modified files and deleted files
> don't have to be propgotten. I suppose this set of pre-commit scripts
> can use the same optimization, so it only scales with the amount of
> added and prop-changed files/dirs in the transaction.
>
> But even then, someday a user comes along that wants to add 60.000
> files (really). He's willing to split it up into commits of say 5000
> files each. The pre-commit hook takes approx 2 hours to validate this,
> so he has to set his client-side timeout very high while doing this
> (after first losing a couple of days before he figures out it's time
> to contact an admin).

Thanks for giving some hard performance numbers and
your approach to mitigate bottlenecks. As you know, I'm
constantly working on eliminating those ...
> Maybe some of this is a moot point, because:
>
> - svn+FSFS+NFS might have gotten much faster at getting props??

Random thought: is it possible to link the transactions
folder to /dev/shm even if the repository is on NFS?
> - maybe propgetting through one of the bindings is much faster than
> through svnlook, and you intend for admins to set this up with some
> language bindings? In that case, keep in mind that bindings are not
> always installed. Svnlook usually is always there.

The problem is probably that svnlook is at a hefty
disadvantage over e.g. the server because it has
to open the FSFS repo anew for each access. Don't
know really what to do about that.
> - maybe you just expect the infrastructure to be much faster?
>
>
> The problem is mainly that 'svnlook propget' doesn't support recursive
> propgetting, nor getting all props from the entire transation, or ...
> only the props for a single path. Which means you have to invoke it
> separately for every item that needs to be inspected. Maybe that can
> be improved somehow?
>
> Anyway, these are just some slightly OT random thoughts.
>
Still a good post ;)

-- Stefan^2.