httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Trawick <traw...@gmail.com>
Subject Re: Finding memory leaks in httpd and httpd modules
Date Wed, 17 Feb 2010 14:12:03 GMT
On Wed, Feb 17, 2010 at 8:30 AM, harish kulkarni <wasinapple@gmail.com> wrote:
> valgrind won't help because apache doesn't free the memory from it's pool,
> so valgrind doesn't catch any.
>
> you look for setting the debug level such that you get to track the pool
> growth.

Are you referring to building APR specially for pool debugging?  That
might allow tools that track heap use a better handle on activity.  (I
haven't tried that in a long time personally).

>
> there is a global pool, there is pool per connection and there is a pool per
> request.

It's a good bit more complicated than that unfortunately.

--/--

I'd love to hear of better practices than what I've used recently,
which is just:

a. get the server to steady state

where steady state = the point at which httpd has allocated any retained memory

easier said than done, since per-thread memory managed by httpd/apr
will continue to grow until every thread has processed the kind of
connection/request that requires the most memory

configure httpd for a fixed set of child processes (for threaded MPMs,
set MaxSpareThreads=MaxClients and MaxRequestsPerChild=0); for a
threaded MPM, one child process is probably best

apply load such that the type of request under consideration has been
handled by every thread

b. see what causes the heap to expand (brk/sbrk)

generally, get a backtrace from brk

DTrace one-liner: /usr/sbin/dtrace -n 'syscall::brk:entry /execname ==
"httpd"/ {ustack();}'

you could also attach a debugger to a running process, set a
breakpoint in brk, and get backtraces manually once brk is called

The indirect caller of brk isn't necessarily the problem, but it is
worth looking at, especially if most calls to brk are for the same
backtrace.

Maybe the backtrace shows questionable pool use -- e.g., some code
allocating from pconf or other pool not tied to the connection.  That
could indicate an essentially broken design for this aspect of the
module driving the allocation, and often appears in combination with a
thread-safety problem.

Maybe the backtrace shows some third-party library code used by
in-process application logic; that might mean the "objects" created by
that library code aren't managed properly.

More likely the backtrace will be a seemingly-normal path and when the
code looks fine at first inspection you'll wonder why this path always
the unlucky one (no magic answer there).  (Even if this code isn't at
fault for the leak and is just the first to need significant memory
after a leak, sometimes a study of this particular code shows that it
is greedier than it could/should be about heap use.)

--/--

I'm hoping to be "embarrassed" by much better hints from the crowd!!!

Mime
View raw message