httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Snow <jsno...@gatesec.net>
Subject Re: [users@httpd] Why does Apache use up all my memory?
Date Thu, 11 Aug 2005 16:40:33 GMT
George,

I have something similar...

I have been debugging an issue where I have seen processes growing to 800Mb on 
a forward proxy configuration using the worker model. Perhaps interestingly 
on reverse proxy configurations I get 100% CPU states occasionally as well. 
What I have noticed is that these are almost always associated with the TCP 
state condition of CLOSE_WAIT. What is netstat saying on your system when the 
problems occur?

Just today I had a win where I was able to truss the process on a forward 
proxy reading data from a server but had nowhere to write as the socket to 
the client was closed/half closed. The error returned (EPIPE) was not being 
caught by apache so there was this continual read then failure to write. 
While this process did not result in a large increase of memory or CPU or 
CLOSE_WAIT states it did verify for me something I had suspected for a long 
time, that the apache code is not checking it's socket calls at some point 
(in this case writev), and/or does not catch a close on the socket. This 
trace was on an ftp connection so the results may be different for an http 
connection e.g. memory usage.

I will be discussing these issues on the dev mailing list in the near future  
but it would be good to see if we are seeing anything in common first.

My hunch (which is a very long one) has been for a while now that when the 
client breaks the connection, the process somehow misses the close and 
continues to read but cannot write as verified by the above. It appears that 
in some instances the memory consumption may be caused by the buffering of 
data or bad memory management which is due to socket issues i.e. I have 
noticed that the larger the download file such as an iso image the larger the 
process grows. As the processes handle additional connections the memory is 
not freed and it grows. Note at this point it is speculation on my part but I 
consistantly get these stateless connections in conjunction with high CPU or 
memory usage so something is going on. I have noticed that processes will 
always creep in size with time so I believe there may be memory leaks but the 
cause of large memory consumption may be caused by weird socket states.

If your clients are downloading 18Mb files over slow links they may keep 
trying the connection, breaking the original therefore leaving you with 
multiple connections to the same file from the same client. If there is a 
memory leak under half close conditions your process grows and your processes 
will handle 5000 connections before they are cycled as per your 
configuration.

But why does this not appear to affect many other people? Firewalls perhaps, 
do you have any network or host based firewalls which may be preventing 
proper shutdown of connections? If so do they drop or reject packets? Which 
firewalls are they? I work in an Internet Gateway environment so I have 
firewalls all over the place and have added these as a variable to my list of 
possibilities.

Your 20 concurrent connections are limited by MaxClients. I assume you are 
keeping this small because of the size they are growing to as you should be 
able to get to approx 175-200 of the top of my head using prefork with 1Gb of 
memory. I would have thought this would max out pretty quickly with many 18Mb 
downloads as they will take time.

As a workaround you may try to lower the MaxRequestsPerChild to turnover 
processes which may be affected by memory leakage and raise the MaxClients to 
handle more concurrent connections. Say initially MaxClients 150 and 
MaxRequestsPerChild 100 or more agressively 10. This will produce more CPU 
overhead in forking processes but modern CPUs are pretty fast. Or go for a 
threaded model such as worker and you should be able to get 10-15 times as 
many concurrent connections (based on proxy configurations - I have never 
used apache as a web server). But another model may simply have the same 
issues if it is socket related. Funnily I was considering going to a prefork 
model to eliminate the possibility of threading and mutex issues - won't be 
doing that for a while.

This may not help but I would be interested in whether there are similarities 
in network states or hardware etc. to what I have.

Regards,
Jon

On Wednesday 10 August 2005 01:05, George Adams wrote:
> I read an earlier thread on memory consumption (http://tinyurl.com/bly4d),
> which may be related to my problem... but because of some differences, I'm
> not so sure.  Any help would be appreciated!
>
> I have an Apache 2.0.54 server on a Gentoo Linux (2.6.11) box which has
> 1Gig RAM and an additional 1Gig swap space.  The server handles a lot of
> people downloading sermons from our church website (which are no larger
> than 18Meg MP3 files), but I can't figure out how to keep the server from
> running out of memory.
>
> Here's my Apache2 prefork configuration:
> ------------------------------------------------------------------------
> <IfModule prefork.c>
>     StartServers         5
>     MinSpareServers      5
>     MaxSpareServers     10
>     MaxClients          20
>     MaxRequestsPerChild  5000
> </IfModule>
>
>
> And here's what the Apache "/server-status" URL showed earlier today (I had
> just restarted the server, but it immediately filled up with download
> requests, all from the same guy, apparently using a download accelerator
> judging by the duplicate requests):
> ------------------------------------------------------------------------
> Srv      PID     M     CPU            Req     Request
> 0-0    15822    W     0.48             0    GET /out/181.mp3 HTTP/1.1
> 1-0    15823    W     0.00    1742573500    GET /out/388.mp3 HTTP/1.1
> 2-0    15824    W     0.00    1742573499    GET /out/238.mp3 HTTP/1.1
> 3-0    15825    W     0.00    1742573499    GET /out/504.mp3 HTTP/1.1
> 4-0    15826    W     0.00    1742573496    GET /out/388.mp3 HTTP/1.1
> 5-0    15832    W     0.00    1742572495    GET /out/801.mp3 HTTP/1.1
> 6-0    15834    W     0.00    1742571493    GET /out/504.mp3 HTTP/1.1
> 7-0    15835    W     0.00    1742571489    GET /out/504.mp3 HTTP/1.1
> 8-0    15838    W     0.00    1742570476    GET /out/388.mp3 HTTP/1.1
> 9-0    15839    W     0.00    1742570484    GET /out/504.mp3 HTTP/1.1
> 10-0    15840    W     0.60             0    GET /out/238.mp3 HTTP/1.1
> 11-0    15841    W     0.00    1742570477    GET /out/388.mp3 HTTP/1.1
> 12-0    15846    W     0.25             0    GET /out/181.mp3 HTTP/1.1
> 13-0    15847    W     0.00    1742569347    GET /out/181.mp3 HTTP/1.1
> 14-0    15848    W     0.00    1742568761    GET /out/801.mp3 HTTP/1.1
> 15-0    15849    W     0.00    1742568761    GET /out/801.mp3 HTTP/1.1
> 16-0    15852    W     0.19             0    GET /out/181.mp3 HTTP/1.1
> 17-0    15853    W     0.17             0    GET /out/801.mp3 HTTP/1.1
> 18-0    15854    W     0.22             0    GET /out/504.mp3 HTTP/1.1
> 19-0    15855    W     0.28             0    GET /server-status HTTP/1.1
>
>
> And here's a portion of what "top" showed at the same time:
> ------------------------------------------------------------------------
> top - 18:09:59 up 64 days,  7:08,  3 users, load avg: 21.62, 10.57, 4.70
> Tasks: 154 total,   1 running, 143 sleeping,   1 stopped,   9 zombie
> Cpu(s):  0.8% us,  2.3% sy, 0.0% ni, 0.0% id, 96.3% wa, 0.3% hi, 0.2% si
> Mem:   1034276k total,  1021772k used,    12504k free,     6004k buffers
> Swap:  1030316k total,   985832k used,    44484k free,    83812k cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 15846 apache    16   0  132m  89m 1968 S  0.3  8.9   0:01.46 apache2
> 15840 apache    17   0  130m  83m 2008 D  0.0  8.3   0:00.90 apache2
> 15849 apache    16   0  120m  82m 1968 S  0.3  8.1   0:01.02 apache2
> 15852 apache    16   0  120m  81m 1968 S  0.3  8.1   0:00.91 apache2
> 15848 apache    16   0  109m  73m 2008 S  0.3  7.2   0:00.85 apache2
> 15855 apache    16   0  107m  70m 2076 D  0.3  7.0   0:00.76 apache2
> 15822 apache    17   0  179m  55m 1968 D  0.3  5.5   0:00.88 apache2
> 15854 apache    16   0 98024  55m 1968 D  0.0  5.5   0:00.58 apache2
> 15853 apache    18   0 98.9m  53m 2000 S  0.0  5.3   0:00.51 apache2
> 15847 apache    17   0 86884  52m 1968 D  0.0  5.2   0:00.42 apache2
> 15841 apache    17   0  110m  36m 1964 D  0.3  3.6   0:00.64 apache2
> 15826 apache    17   0  173m  20m 1968 D  0.0  2.0   0:00.57 apache2
> 15825 apache    16   0 97.7m  19m 1968 D  0.0  1.9   0:00.36 apache2
> 15834 apache    16   0  117m  14m 1968 D  0.3  1.5   0:00.42 apache2
> 15839 apache    17   0  115m  12m 1968 D  0.0  1.2   0:00.40 apache2
> 15838 apache    15   0  182m  12m 1968 D  0.0  1.2   0:00.59 apache2
> 15823 apache    16   0  180m  11m 1968 D  0.0  1.1   0:00.65 apache2
> 15824 apache    15   0  103m 9980 1968 D  0.0  1.0   0:00.27 apache2
> 15832 apache    16   0  116m 9112 1968 D  0.0  0.9   0:00.29 apache2
> 15835 apache    16   0  162m 8844 1968 D  0.0  0.9   0:00.41 apache2
> (everything else listed on "top" below this was less than 0.5 for %MEM)
>
>
> The memory usage swelled very fast as the download requests came in, and
> based on previous experience, the server would have slowed to a crawl and
> possible crashed as it tried to save itself if I hadn't run "killall
> apache2" at this point.
>
> So it seems like this guy's 19 download requests are enough to pretty much
> exhaust my 1 Gig of physical RAM and 1 Gig of swap space.  That just
> doesn't seem right.  EVEN IF something weird was happening where every
> Apache child loaded an entire MP3 file into RAM before serving it, that
> still only accounts for 20 servers * 18Meg files = 360Meg RAM - a lot, but
> nowhere near 2 Gig.  Yet these 20 processes have consumed almost 2 Gig.
>
> What am I doing wrong that so few download requests can bring the server to
> its knees?  How can I fix this configuration?
>
> Thanks to anyone who can help!
>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message