subversion-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Harold <thomas-li...@nybeta.com>
Subject Re: svn 1.8 migration - directory deltification and revprop packing
Date Fri, 02 Aug 2013 19:21:41 GMT
On 6/11/2013 8:52 AM, C. Michael Pilato wrote:
> One advantage of being in a room full of Subversion developers, specifically
> the guy that implemented all this stuff, is that I can ask him directly
> about how to respond to this mail.  :-)  Hopefully I will accurately
> represent the answers Stefan Fuhrmann just gave me to your questions.
>
> On 06/10/2013 03:05 PM, Thomas Harold wrote:
>> a) Why are directory/property deltifications turned off by default?
>
> Stefan's jovial first answer was, "Because I'm a chicken."  :-)
>
> Fortunately, he didn't stop there.  He explained that there is no known
> correctness risk -- you're not going to damage your repositories by enabling
> the feature.  But he wanted to phase in the feature to allow time to collect
> real-world data about the amount of space savings folks are actually
> observing in their repositories.  The feature is on by default in his
> proposed next version of the FSFS format.
>
>> b) Is there a global setting file that can be used to enable
>> directory/property deltifications? Or will we have to update the fsfs.conf
>> file for each newly created repository in order to get this feature?
>
> In 1.8, you'll need to toggle this for each new repository manually.

So in order to get full use of the "Directory and property storage 
reduction" in a retroactive manner as noted in:

http://subversion.apache.org/docs/release-notes/1.8.html#fsfs-enhancements

We will need to:

1. svnadmin dump oldreponame

2. svnadmin create newreponame

3. Modify db/fsfs.conf

[rep-sharing]
enable-rep-sharing = true # defaults to true in 1.8

[deltification]
enable-dir-deltification = true
enable-props-deltification = true

This can be done in automated fashion with sed (or awk).  For example, 
to change compress-packed-revprops:

sed 's/^[#[:space:]]*compress-packed-revprops = 
false[#[:space:]]*$/compress-packed-revprops = true/g' --in-place=bkp 
newreponame/db/fsfs.conf

Or to enable deltification (as well as enable-rep-sharing) all at once:

sed 's/^[#[:space:]]*enable-rep-sharing = 
false[#[:space:]]*$/enable-rep-sharing = 
true/g;s/^[#[:space:]]*enable-dir-deltification = 
false[#[:space:]]*$/enable-dir-deltification = 
true/g;s/^[#[:space:]]*enable-props-deltification = 
false[#[:space:]]*$/enable-props-deltification = true/g' --in-place=bkp 
newreponame/db/fsfs.conf

Naturally, you should have a backup of your db/fsfs.conf file.  While 
sed creates one with --in-place=bkp, you should not rely solely on that 
method.

4. svnadmin load newreponame

5. svnadmin pack newreponame
- Not all of our repos were packed

Even without turning on deltification, I'm seeing a size difference 
between our 1.7 repositories and the ones that we loaded in from dump 
files in 1.8.1.  Possibly from enable-rep-sharing now being set to 
'true' by default in 1.8.

Test repo #1 went from 2.0G to 1.7G (85% of original).

Test repo #2 went from 2.4G down to 2.1G (88% of original).  File count 
in repo #2's fsfs folder only went from 3808 to about 3684 (1820 
revisions).  Even though we did "svnadmin load" using 1.8.1, I still had 
to "svnadmin pack" to cut the file count down to 1692.

Test repo #3 is 10202MB with 47199 files and 45990 revisions (in 1.6 / 
1.7). The revs had been packed, but not the revprops under the old 
system.  The gzip'd dump file is 35.8GB.  I enabled the two 
deltification options in db/fsfs.conf before doing the "svnadmin load" 
step.  Finished size was 7330MB (72% of original) with 92108 files. 
File count drops to 2380 after packing and repository size drops to 
7072MB (69% of original).

Mime
View raw message