Mailing-List: contact dev-help@subversion.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=ymhfrBOUXOommVyTAHOTRubrJnClmKAzbHq4RP1FU/BS78Vh2o60sOVPGssj4xdyxp/qemDBtsNZBtRzqQuDNSifR66EGQhWroYmD+2aiVYckLW9YofctzKdNvq54vZ0jYHBZSzultPfXHHy9CJ0knkuG2+4mgTT4HdPdPqNN5c=;
References: 
 <CAJjMeYOcU9CR091iMFd4JyAC3NMw+GQFCxJ4kHNuGnPvx43gKQ@mail.gmail.com>
 <1332357542.34175.YahooMailNeo@web161405.mail.bf1.yahoo.com>
 <20120321201136.GE15232@daniel3.local>
 <CABD8fLVrQ4pmst7E=V925GDw7Cu=45NhM1y2vyzV7zTQcvgq0g@mail.gmail.com>
 <20120321210737.GA11249@daniel3.local>
 <1332400521.26161.YahooMailNeo@web161401.mail.bf1.yahoo.com>
 <20120322143017.GA3696@daniel3.local>
 <1332429520.60561.YahooMailNeo@web161405.mail.bf1.yahoo.com>
 <CAHFaGCq-uQmtH6fz7jeQK2P+YTG6x209+NbMRa0q5RWF-kVNsQ@mail.gmail.com>
 <4F6B4734.3020307@apache.org> <20120322155022.GA4827@daniel3.local>
 <4F6B4CC3.90901@apache.org> <4F6EFB51.3060901@e-reka.si>
 <1332758839.59233.YahooMailNeo@web161401.mail.bf1.yahoo.com>
  <1333182675.5491.YahooMailNeo@web161406.mail.bf1.yahoo.com>
 <727D8E16AE957149B447FE368139F2B50D78C8BA@SERVER10>
Message-ID: <1333532345.90915.YahooMailNeo@web161404.mail.bf1.yahoo.com>
Date: Wed, 4 Apr 2012 02:39:05 -0700 (PDT)
From: Ashod Nakashian <ashodnakashian@yahoo.com>
Reply-To: Ashod Nakashian <ashodnakashian@yahoo.com>
Subject: Re: Compressed Pristines (Summary)
To: Markus Schaber <m.schaber@3s-software.com>,
  "julianfoad@btopenworld.com" <julianfoad@btopenworld.com>,
  "mtherieau@gmail.com" <mtherieau@gmail.com>
Cc: Subversion Development <dev@subversion.apache.org>
In-Reply-To: <727D8E16AE957149B447FE368139F2B50D78C8BA@SERVER10>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Combined response inline...=0A=0A> From: Markus Schaber <m.schaber@3s-softw=
are.com>=0A>=0A>First, thanks for your great summary. I'll throw in just my=
 2 cents below.=0A=0AThe pleasure is mine.=0A=0A> From: Markus Schaber <m.s=
chaber@3s-software.com>=0A>=0A>Was any of those tests actually executed on =
a file system supporting something like "block suballocation", "tail mergin=
g" or "tail packing"?=0A=0ANo, not to my knowledge. Mine was on standard in=
stallations of Ubuntu 11.10. And I was trying to calculate the waste on a s=
ystem that *didn't* have them enabled.=0A=0A=0A> From: Markus Schaber <m.sc=
haber@3s-software.com>=0A>=0A>Today, I was rather surprised that my pristin=
e subdir of one of our main projects which contains 726 MB of data has an a=
ctual disk size of 759 MB, which leads to an overhead of less than 4% due t=
o block-size rounding. (According to the Explorer "Properties" dialog of Wi=
n 7 on a NTFS file system.)=0A=0ADid you have NTFS compression enabled?=0A=
=0A=0A> From: Markus Schaber <m.schaber@3s-software.com>=0A>=0A>AFAICS, "mo=
dern" file systems increasingly support that kind of feature[1], so we shou=
ld at least think about how much effort we want to throw at the "packing" p=
art of the problem if it's likely to vanish (or, at least, being drasticall=
y reduced) in the future. =0A=0A[snip]=0A=0A> From: Mark Therieau <mtheriea=
u@gmail.com>=0A>=0A>Another thought would be to pursue a FUSE-like approach=
 similar to scord [1][2]=0A[snip]=0A=0A> From: Julian Foad <julianfoad@btop=
enworld.com>=0A>=0A>1.=A0 Filesystem compression.=0A>=0A>Would you like to =
assess the feasibility of compressing the pristine store by re-mounting the=
 "pristines" subdirectory as a compressed subtree in the operating system's=
 file system?=0A=0ANo :-)=0A=0AThere are two ways to answer this interestin=
g proposition of compressed file-systems. The obvious one is that it isn't =
something SVN can or should control. The file-system and certainly system d=
rivers are up to the user and any requirement or suggestion of tempering wi=
th them is decidedly unwarranted and unexpected from a VCS.=0A=0AThe second=
 is more relevant, however. The user may *still* enable/use these schemes w=
ith or without compressed pristine support. After all, we are only concerne=
d with the pristine store and *not* the working copy. So there is still roo=
m for these technologies, if/when the user feels so inclined to utilize the=
m.=0A=0ASo I'd say there is nothing preventing the user from using them, at=
 their responsibility, and get further gains in disk savings while at the s=
ame time they are markedly out of scope for compressed pristines feature, i=
f not SVN as a system.=0A=0A> From: Markus Schaber <m.schaber@3s-software.c=
om>=0A>=0A>Additionally, the simple and efficient way of storing the pristi=
nes in a SQLite database (one blob per file) also prevents us from exploiti=
ng inter-file redundancies during compression, while adding a packing layer=
 on top of sqlite leads to both high complexity and a large average blob si=
ze, and large blobs are probably more efficiently handled by the FS directl=
y.=0A=0AYes. That's what the proposal I drafted is claiming.=0A=0A=0A> From=
: Markus Schaber <m.schaber@3s-software.com>=0A>=0A>To cut it short: I'll "=
take" whatever solution emerges, but my gut feeling tells me that we should=
 use plain files as containers, instead of using sqlite.=0A>=0A>The other a=
spects (grouping similar files into the same container before compression, =
applying a size limit for containers, and storing uncompressible files in u=
ncompressed containers) are fine as discussed.=0A>=0A>I'll try to run some =
statistics using publicly available projects on an NTFS file system, just f=
or comparision.=0A>=0A=0A=0AThat would be great. Please share your finds.=
=0A=0A=0A> From: Mark Therieau <mtherieau@gmail.com>=0A>=0A>If the full goa=
l is to reduce=A0pressure on the underlying file system in the presence=0A>=
of many large working copies (e.g. one per branch) then=A0duplicate pristin=
e contents,=0A>even with super-awesome compression would not match the spac=
e savings of a=0A>de-duplicated, pristine-aware, copy-on-write=A0file syste=
m.=0A=0AThat's assuming there are many duplicates. This is certainly possib=
le, especially with many branches/tags checked out from the same source. Bu=
t I suspect it's a more common scenario to have a single branch checked out=
 from different repositories. In other words, unless we have solid numbers =
that there is more savings by de-duplication, the working assumption is tha=
t improving a single branch by compression will be more useful to more user=
s. Plus, your suggestion is probably part of the unified pristine store (ak=
a ~/.svn) which is out of scope for compressed pristines.=0A=0A> From: Juli=
an Foad <julianfoad@btopenworld.com>=0A>=0A>The pristine store implementati=
on also needs to provide =0A*uncompressed* copies of the files.=A0 Some of =
the API consumers can and =0Ashould read the data through svn_stream_t; thi=
s is the easy part.=A0 Other API consumers -- primarily those that invoke a=
n external 'diff' tool -- need to be given access to a complete uncompresse=
d file on disk.=0A=0AThis is certainly a -minor- complication we'll have to=
 deal with. It's just a technicality, not a show stopper or a problem per-s=
e. The pristine/tmp folder could be cleaned up via svn cleanup, for example=
, or at different check-points. The worse case scenarios are to either to c=
lutter the disk by too many temp uncompressed pristines or to delete them p=
rematurely and force the user to re-run their last command. These aren't fa=
tal and it's easy to find a middle-ground to handle them.=0A=0A-Ash