subversion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Fuhrmann <stefan.fuhrm...@wandisco.com>
Subject On pool / memory usage debugging
Date Mon, 08 Dec 2014 17:42:38 GMT
This post has been prompted by issue 4531 and r1643834
(http://subversion.tigris.org/issues/show_bug.cgi?id=4531
 http://svn.apache.org/viewvc?view=revision&revision=r1643834).
We had a somewhat similar issue with log -v & friends in
summer this year. Now, I dug into the APR code and this
what I found.

The _debug varieties of the APR pool functions don't use
the allocator nor chunky allocation (8k+ at a time) but plain
malloc / free instead and traces them individually. That means:

* On 64 bits, each alloc has a 4x8=32 bytes extra overhead
  for typical CRT implementations. Memory usage can be
  expected to double for everything that's not a delta window
  or fulltext in SVN.

* Our 4MB limit of unused memory on allocators is ineffective.
  On platforms that use MMAP in APR allocators, that can be
  a significant difference. Everywhere else, it may or may not be.

* Since we interleave allocations from use scratch and result
  pools, we can expect high levels of fragmentation. This will
  most likely prevent memory from being freed after peak usage.
  It may also increase peak memory usage.

IOW, pool debugging is nice for tracing allocations but if you
want to measure memory consumption on the OS side, turn
pool debugging off.

APR pools also replace the usually desired malloc / free
symmetry with a pool create / clear|destroy symmetry. The
individual PALLOC / PCALLOC lines in the pool usage trace
will give you context but the things one is interested in are
CREATE, CLEAR and DESTROY lines.

Finally, to minimize cache usage, make sure to disable fulltext
caching as well (enabled by default in 1.9) and set the cache
size to *1*, not 0. The latter would fall back to 1.6-style caches,
keeping a fixed number of objects stored in the svn_fs_t, e.g.
8192 full directories. With ra_serf, a single large dir higher up
in the tree is sufficient to blow get up to 100+ MB because it will
be requested in 100s or 1000s of different revisions.

Membuffer cache, OTOH, does not allocate memory after its
creation. All cache-related allocations use the scratch and
result pools provided by the caller to (de-)serialize cached
data. The cache memory itself is fixed and objects that are
too large to fit in will simply be ignored.

-- Stefan^2.

Mime
View raw message