Mailing-List: contact dev-help@subversion.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAP_GPNjw2c-ahp+zD=nmHdkYwV3PBVuea4VFSpkMYVPZnOOfSA@mail.gmail.com>
References: <CA+298Uhy6TvBJ9hF=BCvViiyAQbgkj4Zwe-0DfCM3=zZL-r_6g@mail.gmail.com>
 <CAP_GPNjw2c-ahp+zD=nmHdkYwV3PBVuea4VFSpkMYVPZnOOfSA@mail.gmail.com>
From: Johan Corveleyn <jcorvel@gmail.com>
Date: Fri, 14 Jul 2017 13:38:08 +0200
Message-ID: <CAB84uBXtdqfsFN=yP+pkowdv2g-cH1AJ-f7Tm2P3PwTxTBpnag@mail.gmail.com>
Subject: Re: Proposal: new fsfs.conf properties
To: Evgeny Kotkov <evgeny.kotkov@visualsvn.com>, Paul Hammant <paul@hammant.org>
Cc: Subversion Development <dev@subversion.apache.org>
Content-Type: text/plain; charset="UTF-8"
archived-at: Fri, 14 Jul 2017 11:38:38 -0000

On Fri, Jul 14, 2017 at 1:18 PM, Evgeny Kotkov
<evgeny.kotkov@visualsvn.com> wrote:
> Paul Hammant <paul@hammant.org> writes:
>
>> 1. compression-exempt-suffixes = mp3,mp4,jpeg
>>
>> 2. deltification-exempt-suffixes = mp3,mp4,jpeg
>>
>> Regardless of the setting of 'compression-level', #1 above two mean certain
>> things can skip the compression attempt.  It must give up at a certain point
>> right?
>
> Hi everyone,
>
> To improve the situation with slow commits of large binary and, possibly,
> incompressible files I committed a patch (http://svn.apache.org/r1801940)
> that adds initial support for LZ4 compression in the backend.  While still
> providing a decent compression ratio, LZ4 offers much faster compression
> even than zlib with level=1, and can skip incompressible data chunks.
> (Presumably, LZ4 is used for on-the-fly compression in different file
> systems for these reasons.)
>
> I have seen significant (up to 3 times) speed improvement for svn import
> and commit with large binary files.  Sometimes, using LZ4 compression can
> even outperform a configuration with disabled compression, if the file is
> at least somehow compressible, as it reduces the time required to write
> the data to disk.
>
> Currently, LZ4 compression is enabled if the fsfs.conf file specifies
> compression-level=1, and all other levels still use zlib for compression.
> Right now, the support for LZ4 is only implemented for the file:// protocol,
> but support/negotiation for other protocols can be added later.

Nice!

But how about deltification? Has anyone tried / benchmarked the effect
of turning off deltification (with or without compression), to see
what the effect would be on the commit time? Like I suggested in this
thread yesterday (i.e. set max-deltification-walk to 0 in fsfs.conf --
or perhaps play with both max-deltification-walk and
max-linear-deltification) ...

-- 
Johan