subversion-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Burba <ptbu...@gmail.com>
Subject Re: svn commit: r952493 - in /subversion/trunk/subversion/libsvn_wc: wc-queries.sql wc_db.c
Date Tue, 08 Jun 2010 13:28:29 GMT
On Mon, Jun 7, 2010 at 8:47 PM,  <gstein@apache.org> wrote:
> Author: gstein
> Date: Tue Jun  8 00:47:22 2010
> New Revision: 952493
>
> URL: http://svn.apache.org/viewvc?rev=952493&view=rev
> Log:
> The query that we used to fetch all children in BASE_NODE and WORKING_NODE
> used a UNION between two SELECT statements. The idea was to have SQLite
> remove all duplicates for us in a single query. Unfortunately, this caused
> SQLite to create an ephemeral (temporary) table and place the results of
> each query into that table. It created an index to remove dupliates. Then
> it returned the values in that ephemeral table. For large numbers of
> nodes, the construction of the table and its index becomes very costly.
>
> This change rebuilds gather_children() in wc_db.c to do the duplicate
> removal manually using a hash table. It does some simple scanning straight
> into an array when it knows duplicates cannot exist (one of BASE or
> WORKING is empty).
>
> The performance problem of svn_wc__db_read_children() was first observed
> in issue #3499. The actual performance improvement is untested so far, but
> I'm assuming pburba can pick up this change and try in his scenario.

On Mon, Jun 7, 2010 at 8:53 PM, Greg Stein <gstein@gmail.com> wrote:
> Hey Paul,
>
> Can you try this change on your large-file-count working copies? I
> believe this should fix the performance problems you were seeing.

Greg,

Short Story: Hours to Seconds

Long Story: This does indeed solve the problems I was seeing:

My test repository was our test suite's Greek tree but with 17,000 1KB
files in a single directory:

Prior to r952493, update and status were taking *quite* some time:

  svn st  01:23:33
  svn up  Gave up after an hour (i.e. lasted longer than my lunch).

With your fix in place, performance improves dramatically:

  svn st  00:00:17
  svn up  00:00:11

Paul

P.S. Thanks!  I was nowhere near figuring this out :-\

Mime
View raw message