Return-Path: X-Original-To: apmail-httpd-users-archive@www.apache.org Delivered-To: apmail-httpd-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 35ACC10A02 for ; Fri, 4 Apr 2014 09:39:57 +0000 (UTC) Received: (qmail 88725 invoked by uid 500); 4 Apr 2014 09:39:54 -0000 Delivered-To: apmail-httpd-users-archive@httpd.apache.org Received: (qmail 88358 invoked by uid 500); 4 Apr 2014 09:39:53 -0000 Mailing-List: contact users-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: users@httpd.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list users@httpd.apache.org Received: (qmail 88349 invoked by uid 99); 4 Apr 2014 09:39:52 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Apr 2014 09:39:52 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of icicimov@gmail.com designates 209.85.214.170 as permitted sender) Received: from [209.85.214.170] (HELO mail-ob0-f170.google.com) (209.85.214.170) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Apr 2014 09:39:48 +0000 Received: by mail-ob0-f170.google.com with SMTP id uz6so3241033obc.15 for ; Fri, 04 Apr 2014 02:39:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=U1tmUDvrYNTxMEHMARAClqIxEsdHMK/kaelEsSQVkmI=; b=TYMFHTNHVtzV4sQjNEiPqL8dIhIYma9Qnbnh7Nvld0P+5De5zlsL3BArzuuUQd8YKP AfXUczIxdVJkkpGmXLfS5Zvj+zGoZmhpR1O8lsIWYKSSo2BnsFSvL1FcoBo009YP3Vh+ GVS/SIM+6qIrVvGRQJo9/tLLPgCiTraOGzekLX5oWrAey/FYIFr65LCZbXctEcA+NmuU eYruKYn3a9UYy6XKvpa7dPpFJB+/eYcz2LZPVX4rBaOqPxXcgI+h+VhBh/Ey8irtpfeB HeZA1zqLOTS5cY7xozcE9JkpOm4M+5Exc1MVMnZa7lwCJvjamUUSk9kcJjEkMgcmF0gY XXkg== MIME-Version: 1.0 X-Received: by 10.60.83.234 with SMTP id t10mr17277163oey.4.1396604367654; Fri, 04 Apr 2014 02:39:27 -0700 (PDT) Received: by 10.76.5.146 with HTTP; Fri, 4 Apr 2014 02:39:27 -0700 (PDT) Received: by 10.76.5.146 with HTTP; Fri, 4 Apr 2014 02:39:27 -0700 (PDT) In-Reply-To: <533D6A6C.9000908@christopherschultz.net> References: <533D6A6C.9000908@christopherschultz.net> Date: Fri, 4 Apr 2014 20:39:27 +1100 Message-ID: From: Igor Cicimov To: users Content-Type: multipart/alternative; boundary=089e0116019c39d60f04f6344bf5 X-Virus-Checked: Checked by ClamAV on apache.org Subject: Re: [users@httpd] High CPU Usage in Amazon EC2 --089e0116019c39d60f04f6344bf5 Content-Type: text/plain; charset=ISO-8859-1 On 04/04/2014 1:05 AM, "Christopher Schultz" wrote: > > All, > > I'm having a problem in production I've never seem before. We are > running a pair of AWS EC2 m1.micro web servers where only one of them in > really in service at any given time. The httpd instance serves some > static content and forwards a great deal of traffic via stunnel to a > single back-end Tomcat server using mod_jk 1.2.37. We have been running > under this configuration for several years with no problems. > Enable the stunnel logs maybe they will reveal something? > Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from > Amazon's previous version (I can't remember which one), including the > package-refresh that comes with it for httpd. The current kernel version > is 3.10.34. The current httpd version is 2.2.26. The package name is > "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a > prefork MPM with the following (default) settings: > > StartServers 8 > MinSpareServers 5 > MaxSpareServers 20 > ServerLimit 256 > MaxClients 256 > MaxRequestsPerChild 4000 > > What I can observe is that the CPU load average is rising from the usual > sub-2.0 value to sometimes as high as 70. That's seventy, not > seven-point-oh. > > I see no errors in the log, and httpd doesn't seem to be dropping any > requests... just running very very slowly. > What if you increase the LogLevel to debug? Meaybe jkLogLevel too. > It seems to come in waves: the load will go up, and everything will slow > down, and then we'll get a reprieve. > Whats the memory usage at those times? If you have sysstat installed you can run sar for some stats about disk cpu and memory. > I can see 22 server processes running right this moment, but the load > average has dropped back to 0.05. > > I've enabled ExtendedStatus and it really doesn't look like there is a > huge number of requests being served. Less than 1 req/sec. This is *not* > a high-load server. I can see some of the httpd child processes using > 20% or more of the CPU as reported by 'top'. > > Is there a good way for me to determine what those processes are doing? > As this is a modestly-used server, I can probably enable additional > logging without too much trouble. > > Any help anyone can provide would be very much appreciated. > > Thanks, > -chris > --089e0116019c39d60f04f6344bf5 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable


On 04/04/2014 1:05 AM, "Christopher Schultz" <chris@christopherschultz.net> wrote:=
>
> All,
>
> I'm having a problem in production I've never seem before. We = are
> running a pair of AWS EC2 m1.micro web servers where only one of them = in
> really in service at any given time. The httpd instance serves some > static content and forwards a great deal of traffic via stunnel to a > single back-end Tomcat server using mod_jk 1.2.37. We have been runnin= g
> under this configuration for several years with no problems.
>
Enable the stunnel logs maybe they will reveal something?

> Last weekend, we upgraded our OS to Amazon Linux 2014.0= 3 (32-bit) from
> Amazon's previous version (I can't remember which one), includ= ing the
> package-refresh that comes with it for httpd. The current kernel versi= on
> is 3.10.34. The current httpd version is 2.2.26. The package name is > "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We ar= e using a
> prefork MPM with the following (default) settings:
>
> StartServers =A0 =A0 =A0 8
> MinSpareServers =A0 =A05
> MaxSpareServers =A0 20
> ServerLimit =A0 =A0 =A0256
> MaxClients =A0 =A0 =A0 256
> MaxRequestsPerChild =A04000
>
> What I can observe is that the CPU load average is rising from the usu= al
> sub-2.0 value to sometimes as high as 70. That's seventy, not
> seven-point-oh.
>
> I see no errors in the log, and httpd doesn't seem to be dropping = any
> requests... just running very very slowly.
>
What if you increase the LogLevel to debug? Meaybe jkLogLevel too.

> It seems to come in waves: the load will go up, and eve= rything will slow
> down, and then we'll get a reprieve.
>
Whats the memory usage at those times? If you have sysstat installed you ca= n run sar for some stats about disk cpu and memory.

> I can see 22 server processes running right this moment= , but the load
> average has dropped back to 0.05.
>
> I've enabled ExtendedStatus and it really doesn't look like th= ere is a
> huge number of requests being served. Less than 1 req/sec. This is *no= t*
> a high-load server. I can see some of the httpd child processes using<= br> > 20% or more of the CPU as reported by 'top'.
>
> Is there a good way for me to determine what those processes are doing= ?
> As this is a modestly-used server, I can probably enable additional > logging without too much trouble.
>
> Any help anyone can provide would be very much appreciated.
>
> Thanks,
> -chris
>

--089e0116019c39d60f04f6344bf5--