subversion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Trent Nelson <tr...@snakebite.org>
Subject Re: Identifying branch roots
Date Mon, 10 Oct 2011 15:57:15 GMT
On 07-Oct-11 6:59 AM, Julian Foad wrote:
> On Fri, 2011-10-07 at 11:29 +0100, Julian Foad wrote:
>> Stefan Sperling wrote:
>>> julianfoad wrote:
>>>> +/* This property marks a branch root. Branches with the same value of this
>>>> + * property are mergeable. */
>>>> +#define SVN_PROP_BRANCHING_ROOT "svn:ignore" /* ### should be "svn:branching-root"
*/
>>
>> Hi Stefan. Thanks for picking up on this.
>>
>>> I think your addition of a 'branch root' property is quite a significant
>>> step. Is this really necessary in order to improve the output of
>>> 'svn mergeinfo' or do you have additional steps planned that go beyond
>>> tuning output?
>>
>> Both.  I think knowing whether the (requested) merge source and target
>> are branch roots (and indeed branches of the *same* "project" or tree)
>> is important for improving the output and diagnostics of "svn mergeinfo"
>> and "svn merge" commands.
>>
>> It could of course enable other new behaviours relating to branches, and
>> I don't know what those are yet (apart from trivial UI things like
>> answering "is this a branch?").
>>
>> So I'm working on the idea that it would be useful to have branch roots
>> identifiable by some mechanism, so I'll add "some mechanism" (currently
>> this property, but I'm totally open to a different mechanism such as
>> branch points being defined in a config file) and see what useful
>> behaviours I can come up with.
>>
>>> There has been some discussion about adding a property for this
>>> and similar purposes in the past, see
>>> http://svn.haxx.se/dev/archive-2009-09/0156.shtml
>>> (there are probably more threads about this topic)
>>
>> Yes, and it's time to figure out what we can usefully do with such
>> information and then we'll know exactly what branch configuration
>> information we need and what's a good way to store it.
>>
>> I'll reply to the rest in a further email.
>>
>> - Julian
Welp, I'm never going to get a better lead in than that, so, hi, folks!
Freelance SCM consultant here; used to specialise in ClearQuest, of all
things, but my last two gigs ended up revolving around Subversion.

Specifically, Subversion merges, in the enterprise, and the, uh, quirks
involved.  Each client had different requirements, and thus, the
solution I ended up delivering to each one differed a bit.  The first
solution was neat, and did all kinds of funky ClearQuest integration
and merge validation, but the second one is more applicable to this
discussion, so I'll describe that first.

In essence, it's a hook framework that attempts to enforce Subversion
best-practices by blocking* incoming commits if it detects one or more
of the following:

         (*) Sometimes it'll block, but phrase the error message
             along the lines of "if you *really* want to do this,
             re-try your commit with the phrase 'CONFIRM MULTI-ROOT
             RENAME' somewhere in your commit message".

     TagCopied
     TagRenamed
     TagRemoved
     TagModified
     TagReplaced
     TagSubtreeCopied
     TagSubtreeRenamed
     TagSubtreeRemoved
     TagSubtreeModified
     TagSubtreeReplaced
     MultipleUnknownAndKnownRootsModified
     MixedRootNamesInMultiRootCommit
     MixedRootTypesInMultiRootCommit
     SubversionRepositoryCheckedIn
     MergeinfoAddedToRepositoryRoot
     MergeinfoModifiedOnRepositoryRoot
     SubtreeMergeinfoAdded
     RootMergeinfoRemoved
     DirectoryReplacedDuringMerge
     EmptyMergeinfoCreated
     TagDirectoryCreatedManually
     BranchDirectoryCreatedManually
     BranchRenamedToTrunk
     TrunkRenamedToBranch
     TrunkRenamedToTag
     BranchRenamedToTag
     BranchRenamedOutsideRootBaseDir
     TagSubtreePathRemoved
     RenameAffectsMultipleRoots
     UncleanRenameAffectsMultipleRoots
     MultipleRootsCopied
     UncleanCopy
     FileRemovedFromTag
     CopyKnownRootSubtreeToValidAbsRootPath
     MixedRootsNotClarifiedByExternals
     CopyKnownRootToIncorrectlyNamedRootPath
     CopyKnownRootSubtreeToIncorrectlyNamedRootPath
     RenamedKnownRootToIncorrectlyNamedRootPath
     MixedChangeTypesInMultiRootCommit
     CopyKnownRootToKnownRootSubtree
     UnknownPathCopiedToIncorrectlyNamedNewRootPath
     RenamedKnownRootToKnownRootSubtree
     FileUnchangedAndNoParentCopyOrRename
     DirUnchangedAndNoParentCopyOrRename
     EmptyChangeSet
     CopyKnownRootToUnknownPath
     CopyKnownRootSubtreeToInvalidRootPath
     NewRootCreatedByRenamingUnknownPath
     UnknownPathCopiedToKnownRootSubtree
     NewRootCreatedByCopyingUnknownPath
     PathCopiedFromOutsideRootDuringNonMerge
     UnknownDirReplacedViaCopyDuringNonMerge
     DirReplacedViaCopyDuringNonMerge
     DirectoryReplacedDuringNonMerge
     PreviousPathNotMatchedToPathsInMergeinfo
     PreviousRevDiffersFromParentCopiedFromRev
     PreviousPathDiffersFromParentCopiedFromPath
     PreviousRevDiffersFromParentRenamedFromRev
     PreviousPathDiffersFromParentRenamedFromPath
     KnownRootPathReplacedViaCopy
     BranchesDirShouldBeCreatedManuallyNotCopied
     TagsDirShouldBeCreatedManuallyNotCopied
     CopiedFromPathNotMatchedToPathsInMergeinfo
     InvariantViolatedModifyContainsMismatchedPreviousPath
     InvariantViolatedModifyContainsMismatchedPreviousRev
     InvariantViolatedCopyNewPathInRootsButNotReplace
     MultipleRootsAffectedByRemove
     AbsoluteRootOfRepositoryCopied
     PropertyChangedButOldAndNewValuesAreSame
     CopiedOrRenamedUnknownPathToIncorrectlyNamedNewRootPath
     UnknownPathRenamedViaReplaceToExistingKnownRoot
     UnknownPathCopiedViaReplaceToExistingKnownRoot
     UnknownPathRenamedToKnownRootSubtree
     UnknownPathCopiedToKnownRootSubtree
     KnownRootSubtreeRenamedViaReplaceToExistingKnownRoot
     UncleanRenameOfRootAncestorPath
     RenamedKnownRootViaReplaceToExistingKnownRoot
     RootPathAncestorRenamedViaReplaceToExistingKnownRoot
     RenamedKnownRootViaReplaceToRootAncestorPath
     RenamedKnownRootViaReplaceToRootAncestorPath
     RootPathAncestorRenamedToValidAbsoluteRootPath
     RootPathAncestorRenamedToValidRootPathSubtree
     RootPathAncestorRenamedToKnownRootSubtree
     RootPathAncestorRenamedViaReplaceToRootAncestorPath
     RenamedKnownRootToUnknownPath
     RenamedKnownRootSubtreeToUnknownPath
     RenamedKnownRootSubtreeToValidRootPath
     RenamedKnownRootSubtreeToIncorrectlyNamedRootPath
     UncleanRename
     RenameRelocatedPathOutsideKnownRoot

         (There's probably room for another e-mail thread just
          discussing all of these conditions; let's just say,
          Subversion repositories in the enterprise rarely look
          like their usually-well-laid out open source repository
          brethren.  What was the Blade Runner line?  "I've seen
          things you people wouldn't believe."? ;-)  My personal
          favorite: 'SubversionRepositoryCheckedIn'.)

So, as you can see, most of these conditions involve the concept of a
root.  Thus, the ability to accurately discern what constitutes a root
took up a large portion of my time.

Hard-coding regexes and forcing all repositories to confirm to a pre-
defined repository layout worked like a charm for my first client, as
I was coming in before they had any Subversion repositories rolled out
into production.  (Well, sort of.)

That unfortunately wasn't feasible for my second client.  They were a
*huge* Subversion shop.  At the time I came in they had something like
960 production repositories, and I wouldn't be surprised if they were
well over 1,000 by now.  There was no standard layout between repos,
and a lot of repos used non-standard branches/tags/trunks paths so
trying to manage 'root detection' via regexes was a non-starter.

For example, a number of repos had layouts like this:

     /foo/trunk
     /foo/branches/1.0.x
     /foo/branches/bugzilla/1081

i.e. 'bugzilla' was just some random directory they created to hold
developer branches related to bugs.  A regex approach would have
matched 'bugzilla' as the branch root, whereas, in fact, the branch
root would have been 1081.

The other non-starter was requiring the admin staff to have to go in
and manually specify what constituted a branch, i.e. setting a 'branch
root' property on relevant paths.  The overhead that would have been
required to do that for ~1,000 repositories (with hundreds, if not
thousands of differently named branches/roots (i.e. not particularly
easy to automate reliably)) was not acceptable (for many enterprisey
reasons mainly surrounding cost).

So, I needed to design the branch detection logic in such a way that
it didn't require any hand-holding from the admins or support staff.

It took two attempts.

For the first attempt, I played around with the notion of a root *base*
directory, i.e. /branches and /tags.  The first thing the framework
would do when processing a pre-commit was create a 'RepositoryRoots'
class (the framework was written in Python FWIW), which would recurse
through the repo up to N-levels deep in order to determine the valid
root base directories.  Except for trunk, which was special, if a
directory had subdirectories that were created by copying another path
(i.e. how tags or branches are created), then the directory would be
considered a root base dir.

That lasted... about a day or two.  It was a leaky abstraction at best,
and broke when I encountered repos with the more non-standard layouts.
(I'm not even sure if I've described it accurately above; but eh, who
cares, it's gone now.)

The problem with the regex and base-root-dir discovery approaches was
that they were essentially heuristic based.  "This directory features
lots of subdirectories that were copies of other paths, therefore, it's
a good chance it's a valid root base directory."

In most cases, yes, that was a valid assumption, but not always.  The
root detection logic was the most critical piece of my solution -- I
wasn't getting paid to correctly detect roots 70% of the time in 60% of
the repos.  It needed to be 100% in 100%.

So, I thought to myself, how can I correctly and autonomously identify
a root with 100% accuracy?  What one property did valid roots share
that I could interrogate?  Heck, what even constitutes a root? A branch
is a root, so is a tag, so is trunk.

....and then it dawned on me.  It seems so simple now, in retrospect:

     In the beginning, there was one root: trunk.  Then it was copied
     elsewhere, and became a branch, or maybe a tag.  These copies are
     also roots, and copies of them should also be considered roots.

Ah, so simple!  I just need to start at revision 0 and work my way up to
HEAD, whilst keeping a record of roots I encounter along the way.  And
that's pretty much it ;-)

Turns out, that approach has worked surprisingly well.  It's been in
production at the second client's site for nearly a year now.  They just
run the 'repo analysis' part of the code against new repositories before
enabling the hooks, and wallah, they get instant root detection and
prevention of some 80-something erroneous conditions.

Here are some techie' details about the implementation.  So, the script
stores root information in a revision property called 'evn:roots' (set
against the root of the repository).  The value of evn:roots at any
given revision will list all of the known roots in the repo at that
revision:

% svn pg --revprop -r26503 evn:roots svn://client.com/repos/foo
{'/build/branches/3.0.1/': {'created': 22323},
  '/build/branches/3.0.2/': {'created': 23129},
  '/build/branches/3.1.0/': {'created': 25804},
  '/build/branches/cvs/0.0.1/': {'created': 26389},
  '/build/branches/bugzilla/4144/': {'created': 22121},
  '/build/branches/bugzilla/6952/': {'created': 17661},
  '/build/release//3.0.0/': {'created': 20774},
  '/build/release/paris/3.0.0/': {'created': 20307},
  '/build/release/rome/3.0.1/': {'created': 22473},
  '/build/trunk/': {'created': 2919},
  '/src/trunk/': {'created': 9353},
  ...

The 'created' revision refers to the revision that the root was created
in.  That's important, 'cause we store special metadata against the root
in the revprop for the revision it was created in:

% svn pg --revprop -r9353 svn://client.com/repos/foo
  ...
  '/src/trunk/': {
     'copies': {
         9834:  [('/src/branches/2.1/', 9835)],
         9997:  [('/src/branches/bugzilla/2800/', 9998)],
         10211: [('/src/branches/bugzilla/3326/', 10212)],
         10252: [('/src/branches/bugzilla/2160/', 10253)],
         10468: [('/src/branches/2.2/', 10469)],
         11148: [('/src/branches/2.3/', 11149)],
         11420: [('/src/branches/bugzilla/3720/', 11421)]},
     'created': 9353,
     'creation_method': 'created'},
  ...

i.e. we store all the subsequent forward-copies of this root, as well as
details of how it was created (which isn't very interesting in this
example, as it's trunk and was created via mkdir, but if it were a
branch or tag, it would contain details about where it was copied from).

Let's say I delete /src/trunk in r26504.  The entry for it in evn:roots
in that revision will be gone; but a note will be made against the r9353
creation revprop to indicate which rev it was deleted in.

The importance of storing data like this becomes apparent when you deal
with situations like this:

  *hooks are turned off*
     r2:     svn cp ^/trunk ^/branches/foo
     r3:     svn rm ^/branches/foo
     r4:     svn mkdir ^/branches/foo
  *repo is analysed, evn:roots are set, hooks are turned on*

An attempt to do the following would be blocked, because r4/HEAD of
/branches/foo was not created correctly (i.e. wasn't copied from an
existing root), and thus, isn't considered a root either:

     svn cp /branches/foo /branches/bar

However, the following *would* work, because /branches/foo *was* a valid
root in r2:

     svn cp -r2 /branches/foo /branches/bar


Thoughts?

     Trent.


Mime
View raw message