Mailing-List: contact users-help@subversion.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (athena.apache.org: local policy)
From: Bob Archer <Bob.Archer@amsi.com>
To: Mark Phippard <markphip@gmail.com>, Geoff Rowell <geoff.rowell@gmail.com>
CC: Nick <nospam@codesniffer.com>, "users@subversion.apache.org"
	<users@subversion.apache.org>
Date: Wed, 2 Feb 2011 10:36:44 -0500
Subject: RE: Checkout really slow in Windows with lots of files in one
 directory
Thread-Topic: Checkout really slow in Windows with lots of files in one
 directory
Thread-Index: AcvC2Brl0YDYSJjRTwGgwNMPJyckMwAFR3Iw
Message-ID: <CB4587DA62024B40B9F929ECF639210F2C7687DA32@USALWEXMB4.infor.com>
References: <4D402FA4.4020607@jibbyjobby.co.uk>
	<AANLkTinCzLKz7PJe3yutvhewP0JyyMV58TMaB8RvZSe+@mail.gmail.com>
	<1296637783.6917.192.camel@nimble.325Bayport>
	<AANLkTim5HccMh1LvQgu=Ra+Au7h=Y=8h1MYd4D5CEJP6@mail.gmail.com>
 <AANLkTinhY0BDCMUpTO2rssWzMLjjxvskkN+LPiHN+k6o@mail.gmail.com>
In-Reply-To: <AANLkTinhY0BDCMUpTO2rssWzMLjjxvskkN+LPiHN+k6o@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

> On Wed, Feb 2, 2011 at 7:41 AM, Geoff Rowell
> <geoff.rowell@gmail.com> wrote:
> > On Wed, Feb 2, 2011 at 4:09 AM, Nick <nospam@codesniffer.com>
> wrote:
> >> On Tue, 2011-02-01 at 13:00 -0500, Mark Phippard wrote:
> >>
> >> On Wed, Jan 26, 2011 at 9:28 AM, Neil Bird
> <neil@jibbyjobby.co.uk> wrote:
> >>
> >>> =A0We have a graphics-oriented code-base that's auto-generated
> and has >5000
> >>> source files in one directory. =A0While I can check this out OK
> on Linux,
> >>> we're seeing an unusable slow-down on Windows XP (NTFS), both
> using
> >>> Tortoise
> >>> directly, and as a test on Linux with the Windows drive mapped
> over CIFS.
> >>
> >> I created a folder with 5001 files in it ... maybe that is not
> enough?
> >> =A0I just used small simple text files as I was only checking for
> the
> >> general problem in managing the temp files and the WC metadata.
> >>
> >> Upon checkout (using 1.6.15 command line client) I did not
> notice any
> >> slowdown. =A0Windows checked out via HTTP across internet in about
> 49
> >> seconds as opposed to 33 from my Mac (which is a faster system).
> =A0The
> >> main thing is checkout did not seem to slow down.
> >>
> >> I did a similar test, using 5100 files in a single directory.
> Each file
> >> contained only the content "file XXXX" where XXXX was the number
> of the file
> >> (so tiny files).=A0 My linux system took 17 seconds, while Windows
> took a bit
> >> less than 2 min (but Windows is virtualized while linux is on
> the
> >> hardware).=A0 I also did not notice a slow-down as the checkout
> proceeded.
> >> Both systems used 1.6.15 and accessed the repo via https.
> >>
> >> I did, however, notice that the time to *add* the files (done
> via svn add
> >> *.txt) seemed to progressively slow down.=A0 But this was only
> observed by
> >> watching the files in the console as they were being added (it
> was
> >> relatively easy to see the rate because the each file name had a
> linear
> >> number at the end).=A0 I don't have any timings to back this up,
> though I'll
> >> collect some if anyone's interested.
> >>
> > I don't know why, but I believe the key thing here is working
> with
> > *binary* files.
> >
> > I noticed the same problem with a massive (10K+) amount of audio
> > snippets in a single directory.
>=20
> I was thinking that this was a case where the
> reading/parsing/writing
> of our large entries file was causing a slowdown and moving to
> SQLite
> was going to bring performance gains.  Clearly that is not the case
> as
> trunk is much slower.
>=20
> If I get another batch of free time I will try it with a lot of
> small PNG's.

Running Working Copies on RAM drives in Windows makes it fly like the devil=
. For anyone inclined to give it a try. Of course, you need some free RAM t=
o be able to do it. I set up a 2GB RAM drive to try it. I was really trying=
 to improve Visual Studio perf though, and not svn... so I don't use it any=
 more. But, I did notice that all svn stuff was much much faster. I can't r=
ecall the software I tried.

BOb