httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: [users@httpd] High CPU Usage in Amazon EC2
Date Thu, 03 Apr 2014 15:40:31 GMT
All,

This is the current status map on me server:

Total accesses: 14118 - Total Traffic: 3.9 GB
CPU Usage: u404.04 s9.82 cu0 cs0 - .956% CPU load
.326 requests/sec - 94.4 kB/second - 289.5 kB/request
6 requests currently being processed, 15 idle workers

.__..._.._.R_R_......_...__.._._R_.R...R..W_..__................
................................................................
................................................................
................................................................

top reports CPU load average "4.71, 10.49, 10.43" so things are settling
down a bit right now. Here's what top shows for the httpd processes (at
the top of the report, so these are the highest-CPU processes this
particular moment):

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

23225 apache    20   0 27772 6860 2444 R  8.6  1.1   0:02.94 httpd

23337 apache    20   0 27764 6724 2324 R  6.3  1.1   0:00.80 httpd

23224 apache    20   0 27820 7112 2624 R  4.6  1.2   0:03.02 httpd

23023 apache    20   0 27824 7204 2692 R  4.3  1.2   0:12.15 httpd

21815 apache    20   0 27820 7420 2916 R  4.0  1.2   0:58.10 httpd

22478 apache    20   0 27992 8412 3856 R  4.0  1.4   0:28.18 httpd


I wasn't able to copy/paste all of that stuff at exactly the same time,
so the numbers are a bit off from each other.

But the server is simply not handling very much traffic. I don't
understand the spikes in CPU load -- which is what lows everything down.
Even doing something like attaching strace to a running process took
something like 30 seconds before it started actually logging anything to
the terminal.

I checked the status of entropy availability using this command:

$ cat /proc/sys/kernel/random/entropy_avail

I did this a few tens of times and the lowest number I ever got was 133.
On systems that have truly run out of randomness, I have seen this
number drop to zero (it usually happens during OpenSSL server key
generation :).

I'm at a loss for what to check next. Any suggestions?

Thanks,
-chris

On 4/3/14, 10:38 AM, Christopher Schultz wrote:
> Oscar,
> 
> On 4/3/14, 10:33 AM, Oscar Knorn wrote:
>> Sounds like apache is waiting for a response or a means to forward the
>> request via stunnel.
>> Is /dev/random or /dev/urandom providing sufficient random to the process?
> 
> I'm not sure how to check that. Any ideas?
> 
> In either case above, wouldn't the processes be sitting idle waiting for
> blocking IO to return? I'm not surprised that the bytes are moving
> slowly, actually. I'm surprised that I'm pegging my CPU waiting for
> bytes to move...
> 
> Thanks,
> -chris
> 
>> On 4/3/14, 40:23 PM, Christopher Schultz wrote:
>>> All,
>>>
>>> I forgot to mention that most of our traffic is over SSL. OpenSSL
>>> version is OpenSSL 1.0.1e-fips 11 Feb 2013.
>>>
>>> Thanks,
>>> -chris
>>>
>>> On 4/3/14, 10:04 AM, Christopher Schultz wrote:
>>>> All,
>>>>
>>>> I'm having a problem in production I've never seem before. We are
>>>> running a pair of AWS EC2 m1.micro web servers where only one of them in
>>>> really in service at any given time. The httpd instance serves some
>>>> static content and forwards a great deal of traffic via stunnel to a
>>>> single back-end Tomcat server using mod_jk 1.2.37. We have been running
>>>> under this configuration for several years with no problems.
>>>>
>>>> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
>>>> Amazon's previous version (I can't remember which one), including the
>>>> package-refresh that comes with it for httpd. The current kernel version
>>>> is 3.10.34. The current httpd version is 2.2.26. The package name is
>>>> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
>>>> prefork MPM with the following (default) settings:
>>>>
>>>> StartServers       8
>>>> MinSpareServers    5
>>>> MaxSpareServers   20
>>>> ServerLimit      256
>>>> MaxClients       256
>>>> MaxRequestsPerChild  4000
>>>>
>>>> What I can observe is that the CPU load average is rising from the usual
>>>> sub-2.0 value to sometimes as high as 70. That's seventy, not
>>>> seven-point-oh.
>>>>
>>>> I see no errors in the log, and httpd doesn't seem to be dropping any
>>>> requests... just running very very slowly.
>>>>
>>>> It seems to come in waves: the load will go up, and everything will slow
>>>> down, and then we'll get a reprieve.
>>>>
>>>> I can see 22 server processes running right this moment, but the load
>>>> average has dropped back to 0.05.
>>>>
>>>> I've enabled ExtendedStatus and it really doesn't look like there is a
>>>> huge number of requests being served. Less than 1 req/sec. This is *not*
>>>> a high-load server. I can see some of the httpd child processes using
>>>> 20% or more of the CPU as reported by 'top'.
>>>>
>>>> Is there a good way for me to determine what those processes are doing?
>>>> As this is a modestly-used server, I can probably enable additional
>>>> logging without too much trouble.
>>>>
>>>> Any help anyone can provide would be very much appreciated.
>>>>
>>>> Thanks,
>>>> -chris
>>>>
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
> 


Mime
View raw message