Mailing-List: contact dev-help@subversion.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <2D4384B1-34FF-42CB-9DD1-1AA6F3A2F6A4@gmail.com>
References: <CAP_GPNhP6sx-noU_dvN8JUeJA=z=3HpF7duzchOpY2AK+QAC1Q@mail.gmail.com>
 <2D4384B1-34FF-42CB-9DD1-1AA6F3A2F6A4@gmail.com>
From: Jacek Materna <jacek@assembla.com>
Date: Fri, 4 Aug 2017 09:35:31 -0500
Message-ID: <CAGjJbctmx2PYXFV6PEDVwD+=N-9zY-FMZbL6-uMHcnLDXxE_Jg@mail.gmail.com>
Subject: Re: [RFC] Using LZ4 compression by default
To: Nathan Hartman <hartman.nathan@gmail.com>
Cc: Subversion Development <dev@subversion.apache.org>
Content-Type: multipart/alternative; boundary="001a113cea722b5c190555ee6b40"
archived-at: Fri, 04 Aug 2017 14:36:24 -0000

--001a113cea722b5c190555ee6b40
Content-Type: text/plain; charset="UTF-8"

Based on what we are seeing - disk storage is at least 10x less concerning
across a huge swatch of enterprise SVN users than UX (performance and speed
of developer workflows). It's becoming less and less so each quarter.
Unless the disk storage increase impacts the server side performance in
aggregate I don't understand why the priority (biased defaults) would not
be to increase the speed of every operation.

?

On Fri, Aug 4, 2017 at 9:30 AM, Nathan Hartman <hartman.nathan@gmail.com>
wrote:

> On Aug 2, 2017, at 2:59 PM, Evgeny Kotkov <evgeny.kotkov@visualsvn.com>
> wrote:
> > With the recently added support for LZ4 compression (r1801940 et al),
> > we now have an option of using it by default for the on-disk data and
> > over the wire.
> > [...]
> > The amount of the required additional space is proportional
> >     to the difference between the compression ratio of LZ4 and zlib-5,
> >     which can be roughly estimated as around 30-35% for compressible
> >     binary and text files, although that may vary depending on the
> >     actual data.
> >
> > To illustrate how these changes will affect the speed of some of the
> > operations, the 'svn import' of a 2 GB file over HTTP on LAN in my
> > environment takes 18 seconds instead of 63 seconds.
>
> Regarding on disk storage, for small repos a 30% size increase is probably
> not material, but it may be significant for large repos. Is it feasible to
> get the best of both worlds by using LZ4 for fast commits and then
> recompress using zlib in svnadmin pack?


-- 

Jacek Materna
Chief Technology Officer

Assembla
+1 210 410 7661

--001a113cea722b5c190555ee6b40
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Based on what we are seeing - disk storage is at least 10x=
 less concerning across a huge swatch of enterprise SVN users than UX (perf=
ormance and speed of developer workflows). It&#39;s becoming less and less =
so each quarter. Unless the disk storage increase impacts the server side p=
erformance in aggregate I don&#39;t understand why the priority (biased def=
aults) would not be to increase the speed of every operation.<div><br></div=
><div>?</div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote=
">On Fri, Aug 4, 2017 at 9:30 AM, Nathan Hartman <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:hartman.nathan@gmail.com" target=3D"_blank">hartman.nathan@gm=
ail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D=
"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=
=3D"">On Aug 2, 2017, at 2:59 PM, Evgeny Kotkov &lt;<a href=3D"mailto:evgen=
y.kotkov@visualsvn.com">evgeny.kotkov@visualsvn.com</a>&gt; wrote:<br>
&gt; With the recently added support for LZ4 compression (r1801940 et al),<=
br>
&gt; we now have an option of using it by default for the on-disk data and<=
br>
&gt; over the wire.<br>
</span>&gt; [...]<br>
<span class=3D"">&gt; The amount of the required additional space is propor=
tional<br>
&gt;=C2=A0 =C2=A0 =C2=A0to the difference between the compression ratio of =
LZ4 and zlib-5,<br>
&gt;=C2=A0 =C2=A0 =C2=A0which can be roughly estimated as around 30-35% for=
 compressible<br>
&gt;=C2=A0 =C2=A0 =C2=A0binary and text files, although that may vary depen=
ding on the<br>
&gt;=C2=A0 =C2=A0 =C2=A0actual data.<br>
&gt;<br>
&gt; To illustrate how these changes will affect the speed of some of the<b=
r>
&gt; operations, the &#39;svn import&#39; of a 2 GB file over HTTP on LAN i=
n my<br>
&gt; environment takes 18 seconds instead of 63 seconds.<br>
<br>
</span>Regarding on disk storage, for small repos a 30% size increase is pr=
obably not material, but it may be significant for large repos. Is it feasi=
ble to get the best of both worlds by using LZ4 for fast commits and then r=
ecompress using zlib in svnadmin pack?</blockquote></div><br><br clear=3D"a=
ll"><div><br></div>-- <br><div class=3D"gmail_signature" data-smartmail=3D"=
gmail_signature"><br>Jacek Materna<br>Chief Technology Officer<br><br>Assem=
bla<br>+1 210 410 7661</div>
</div>

--001a113cea722b5c190555ee6b40--