apr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sami Tolvanen <sami.tolva...@mywot.com>
Subject [PATCH] apr_memcache memory leak with persistent connections
Date Mon, 05 Jan 2009 11:07:42 GMT
Hello all,

We ran into a problem with apr_memcache after upgrading from httpd 2.0
to 2.2. Depending on the TTL passed to apr_memcache_server_create in our
Apache module, we were either leaking lots of memory or grinding to a
halt a few seconds after httpd was started due to an overwhelming number
of connections to memcached servers.

This turned out to be a known problem with apr_memcache, which we didn't
trigger with the apr_reslist implementation used in httpd 2.0 / APR-util
0.9.x. It seems to have been first reported ~two years ago:

  [1] http://issues.outoforder.cc/view.php?id=68
  [2] http://tinyurl.com/apr-dev-200701-memcache
  [3] http://tinyurl.com/apr-dev-200810-memcache

To sum it up, the problem is that memcache functions keep allocating
memory from a connection-specific pool during the entire lifetime of the
connection. Therefore, for a process that constantly performs requests,
any reasonable TTL parameter means there are server connections that are
never closed and just keep consuming more memory.

The patch in [3] seems to mostly mitigate the memory leak issue,
but addresses only apr_memcache_getp and doesn't work for multiline
data. I ended up writing the attached patch (against APR-util 1.3.4),
which allocates bucket brigades from a temporary pool stored in
apr_memcache_conn_t and clears it after each request. We have used this
in production for a couple of days now and the memory leak problem with
persistent connections is gone. There doesn't seem to be any noticeable
performance impact either, at least for our work load.

I would appreciate it if someone more familiar with the memcache code
could review the patch and let me know if there are any gotchas with
this approach. Unless there's a better way for fixing the memory leak
without modifying the API, I propose applying this patch to APR-util as
more than one apr_memcache user seems to have been affected over the
years.

Sami

Mime
View raw message