Return-Path: X-Original-To: apmail-httpd-users-archive@www.apache.org Delivered-To: apmail-httpd-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D92F6110D8 for ; Fri, 22 Aug 2014 11:43:20 +0000 (UTC) Received: (qmail 1188 invoked by uid 500); 22 Aug 2014 11:43:17 -0000 Delivered-To: apmail-httpd-users-archive@httpd.apache.org Received: (qmail 1147 invoked by uid 500); 22 Aug 2014 11:43:17 -0000 Mailing-List: contact users-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: users@httpd.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list users@httpd.apache.org Received: (qmail 1137 invoked by uid 99); 22 Aug 2014 11:43:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Aug 2014 11:43:17 +0000 X-ASF-Spam-Status: No, hits=4.1 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,HTTP_ESCAPED_HOST,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of trawick@gmail.com designates 209.85.215.46 as permitted sender) Received: from [209.85.215.46] (HELO mail-la0-f46.google.com) (209.85.215.46) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Aug 2014 11:43:13 +0000 Received: by mail-la0-f46.google.com with SMTP id b8so10040790lan.33 for ; Fri, 22 Aug 2014 04:42:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=4TvxSOit/zQ9P2RAOSfpDYR40CSesrXv6Mavjedv+3M=; b=07w7mZK54PR91ADxr7hAYCVgFyzt9khk5NemD7+5nevoh6LU1Q6x6KS9LS8YBCVDGx byk+zdzGap0JrvUAndI64PG3jqb7uFZVdKqsht0qxL+ogtPDJGVL5XoA1ERkOCV9Hax6 GPFmPdgcTG9MkeEfSUDZrI9vvDcNiWvOSu070hkot5eIYilZay7wH0FIbejthU5fa7zr 1bf8NWiXCvHv8TtmWQJSrtTGjA+uXXQKC3zjByAOLlUhQcNtmpEMl170kHDI0oYWvkad Yf9nx1mtpWvdtl99N+kkmE56XS054+t9NvukbaAUuM6DLU1bb/6OZxBaSZ3HqhHGhecs GtNw== MIME-Version: 1.0 X-Received: by 10.152.6.133 with SMTP id b5mr4368930laa.16.1408707771409; Fri, 22 Aug 2014 04:42:51 -0700 (PDT) Received: by 10.114.3.20 with HTTP; Fri, 22 Aug 2014 04:42:51 -0700 (PDT) In-Reply-To: <182413CA4B98CC45B193747D004607301F42854B@G9W0768.americas.hpqcorp.net> References: <182413CA4B98CC45B193747D004607301F4282E0@G9W0768.americas.hpqcorp.net> <182413CA4B98CC45B193747D004607301F428415@G9W0768.americas.hpqcorp.net> <182413CA4B98CC45B193747D004607301F42854B@G9W0768.americas.hpqcorp.net> Date: Fri, 22 Aug 2014 07:42:51 -0400 Message-ID: From: Jeff Trawick To: "users@httpd.apache.org" Content-Type: multipart/alternative; boundary=089e01493fa84eabed0501365687 X-Virus-Checked: Checked by ClamAV on apache.org Subject: Re: [users@httpd] MaxRequestsPerChild - New child process doesn't process requests --089e01493fa84eabed0501365687 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Fri, Aug 22, 2014 at 3:46 AM, Vattikuti, Vamsi Krishna Venkata (STSD) < vamsik@hp.com> wrote: > Hi, > > > > Thanks for the feedback. > > > > That tomcat application(monitoring) is invoked through a 3rd party module > and application remains active in browser for days together, which resul= t > in memory leak for httpd process. To prevent that leak, that application > owner has configured MaxRequestsPerChild as 100. I will suggest them to > increase values of this and other directives. > It is likely that you won't find a satisfactory solution as long as you have memory leaks in httpd and extremely-long-running requests. The leak needs to be fixed. I think you will be better off with the prefork MPM until this can be fixed= : * Since you have such a low MaxClients, you're not getting much benefit from the worker MPM. * When a prefork child process reaches MaxRequestsPerChild, it can exit immediately since there are no other active requests in the child process, and not be stranded forever, consuming resources but not able to accept new connections. * If necessary, a person or a monitoring script can kill individual prefork child processes based on time of active request or memory use, and only affect a single active request. If this third-party module also requires exactly one child process, in addition to leaking memory and requiring child process cleanup, you've hit double jeopardy. Switch to another line of work. > > Regards, > > Vamsi. > > > > *From:* Daniel [mailto:dferradal@gmail.com] > *Sent:* Friday, August 22, 2014 12:50 PM > > *To:* users@httpd.apache.org > *Subject:* Re: [users@httpd] MaxRequestsPerChild - New child process > doesn't process requests > > > > What memory leak exactly? You are just proxying to tomcat. > > > > Increase the number of maxrequestsperchild to a more sensible number, suc= h > as 10.000 or even higher, so httpd is not constantly renewing childs if y= ou > have even a minimum load. > > > > Add more servers to avoid your issue, and try to use more threads too, 50 > at least. > > > > It seems like you are constraining too much, and apache 2.2.15 with > mpm_worker can give much much more. > > > > You will notice performance increasing greatly overall. > > > > > > > > 2014-08-22 6:06 GMT+02:00 Vattikuti, Vamsi Krishna Venkata (STSD) < > vamsik@hp.com>: > > Hi Jeff, > > > > Thanks for quick response. We will check and do the needful > > > > That 100 is to prevent memory leak. > > > > Regards, > > Krishna. > > > > *From:* Jeff Trawick [mailto:trawick@gmail.com] > *Sent:* Friday, August 22, 2014 2:24 AM > *To:* users@httpd.apache.org > *Subject:* Re: [users@httpd] MaxRequestsPerChild - New child process > doesn't process requests > > > > On Thu, Aug 21, 2014 at 4:39 PM, Vattikuti, Vamsi Krishna Venkata (STSD) = < > vamsik@hp.com> wrote: > > Hi, > > > > We are having an issue with Tomcat application accessing through proxy an= d > details are below. Can you please check and share your feedback. > > > > *Issue:* > > We have an application(tomcat) accessed through proxy as below. Also, we > have MaxRequestsPerChild setting as 100 > > > > Whenever MaxRequestsPerChild reached its limit, a new process is started > but the application become unresponsive. It seems that new process doesn= =E2=80=99t > service any requests > > We have to restart httpd to recover that > > > > > > *Log:* > > - Access.log doesn=E2=80=99t show any requests for new child > > - Error_log shows that > > a) workers are initiated for new process but it didn=E2=80=99t servi= ce any > requests > > b) > > c) processing has stuck for a minute due to some reason > > [Fri Aug 08 16:09:17 2014] [debug] ssl_engine_kernel.c(2118): [client > 127.0.0.1] Certificate Verification, depth 0 [subject: > /C=3Dy/ST=3Dy/L=3Dy/O=3Dy/OU=3Dy/CN=3Dy, issuer: /C=3Dy/ST=3Dy/L=3Dy/O=3D= y/OU=3Dy/CN=3Dy, serial: > xyz] > > [Fri Aug 08 16:10:12 2014] [info] [client 10.150.90.25] Connection to > child 6 established (server *:) > > > > d) SSL handshake has started but didn=E2=80=99t complete for 4 conne= ctions > related to new process. There are no errors related to ssl > > $ grep -i handshake errorlog.2014-08-08-07_06_44 | grep -c start > > 707 > > $ grep -i handshake errorlog.2014-08-08-07_06_44 | grep -c done > > 703 > > $ > > > > *Apache version:* > > 2.2.15 > > > > > > *Proxy setting:* > > SSLProxyEngine On > > SSLProxyCipherSuite ALL > > SSLProxyMachineCertificateFile /var/ssl/proxy.pem > > > > proxyPass /app1 https://localhost:/app1 (Tomcat) > > > > *Worker configuration:* > > KeepAlive On > > MaxKeepAliveRequests 100 > > KeepAliveTimeout 15 > > > > StartServers 1 > > MaxClients 25 > > MinSpareThreads 12 > > MaxSpareThreads 25 > > ThreadsPerChild 25 > > ServerLimit 1 > > MaxRequestsPerChild 100 > > MaxMemFree 50 > > > > > > Thanks & Regards, > > Krishna > > > MaxRequestsPerChild 100 is ridiculously low. What is happening in httpd > to cause you to need that setting? > > > > Anyway... > > > > Once an httpd child process has reached 100 connections, it initiates a > graceful shutdown, which means that instead of aborting current requests = it > will instead wait for current requests to finish, then exit. > > > > During the time that it is waiting for current requests to finish, new > connections must be handled by other child processes. BUT you set > ServerLimit to 1 (and other directives such as ThreadsPerChild and > MaxClients are consistent with allowing only one child process), so no > other child process can be created during that time. > > > > Thus, once 100 connections are handled, new clients will be blocked until > existing requests finish. > > > > --/-- > > > > My guess: Your Java application takes a long time (maybe forever?) to > handle some requests. MaxRequestsPerChild makes it worse. If the Java > requests are slow and eventually finish, the solution is to keep a steady > set of httpd child processes (having them gracefully exit when there are > slow backend requests can be harmful) and increase the number of httpd > threads/child processes to handle the load. > > > > If some Java requests hang, see how to handle that on the Tomcat side. > > > Enable server status with ExtendedStatus On and watch what happens -- > whether or not certain requests handled by the Java application take a > relatively long time, tieing up some or all of your very limited number o= f > httpd threads. > > > > -- > > Born in Roswell... married an alien... > http://emptyhammock.com/ > > > > > --=20 Born in Roswell... married an alien... http://emptyhammock.com/ --089e01493fa84eabed0501365687 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On F= ri, Aug 22, 2014 at 3:46 AM, Vattikuti, Vamsi Krishna Venkata (STSD) <vamsik@h= p.com> wrote:

Hi,<= /p>

=C2=A0

Thanks for the feedback.<= u>

=C2=A0

That tomcat application(m= onitoring) is invoked through a 3rd party module and =C2=A0appli= cation remains active in browser for days together, which result in memory leak for httpd process. To prevent that leak, that application o= wner has configured MaxRequestsPerChild as 100. I will suggest them to incr= ease values of this and other directives.


It is likely that you won't find a satisfactory sol= ution as long as you have memory leaks in httpd and extremely-long-running = requests. =C2=A0The leak needs to be fixed.

I thin= k you will be better off with the prefork MPM until this can be fixed:

* Since you have such a low MaxClients, you're not = getting much benefit from the worker MPM.
* When a prefork child = process reaches MaxRequestsPerChild, it can exit immediately since there ar= e no other active requests in the child process, and not be stranded foreve= r, consuming resources but not able to accept new connections.
* If necessary, a person or a monitoring script can kill individual pr= efork child processes based on time of active request or memory use, and on= ly affect a single active request.

If this third-p= arty module also requires exactly one child process, in addition to leaking= memory and requiring child process cleanup, you've hit double jeopardy= . =C2=A0Switch to another line of work.

=C2=A0

Regards,

Vamsi.

=C2=A0

From: Daniel= [mailto:dferradal= @gmail.com]
Sent: Friday, August 22, 2014 12:50 PM


To: user= s@httpd.apache.org
Subject: Re: [users@httpd] MaxRequestsPerChild - New child process d= oesn't process requests

=C2=A0

What memory leak exactly? You are just proxying to t= omcat.

=C2=A0

Increase the number of maxrequestsperchild to a more= sensible number, such as 10.000 or even higher, so httpd is not constantly= renewing childs if you have even a minimum load.

=C2=A0

Add more servers to avoid your issue, and try to use= more threads too, 50 at least.

=C2=A0

It seems like you are constraining too much, and apa= che 2.2.15 with mpm_worker can give much much more.

=C2=A0

You will notice performance increasing greatly overa= ll.

=C2=A0

=C2=A0

=C2=A0<= /p>

2014-08-22 6:06 GMT+02:00 Vattikuti, Vamsi Krishna V= enkata (STSD) <vamsik= @hp.com>:

Hi Jeff,=

=C2=A0

Thanks for quick response= . We will check and do the needful

=C2=A0

That 100 is to prevent me= mory leak.

=C2=A0

Regards,=

Krishna.=

=C2=A0

From: Jeff T= rawick [mailto:trawi= ck@gmail.com]
Sent: Friday, August 22, 2014 2:24 AM
To: user= s@httpd.apache.org
Subject: Re: [users@httpd] MaxRequestsPerChild - New child process d= oesn't process requests

=C2=A0

On Thu, Aug 21, 2014 at 4:39 PM, Vattikuti, Vamsi Kr= ishna Venkata (STSD) <vamsik@hp.com> wrote:

Hi,

=C2=A0

We are having an issue with Tomcat application acces= sing through proxy and details are below. Can you please check and share yo= ur feedback.

=C2=A0

Issue:

We have an application(tomcat) accessed through prox= y as below. Also, we have MaxRequestsPerChild setting as 100<= /p>

=C2=A0

Whenever MaxRequestsPerChild reached its limit, a ne= w process is started but the application become unresponsive. It seems that= new process doesn=E2=80=99t service any requests

We have to restart httpd to recover that

=C2=A0

=C2=A0

Log:

-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Access.log doesn=E2=80=99t show any requests for new= child

-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Error_log shows that

a)=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 workers are initiated for new process but it d= idn=E2=80=99t service any requests

b)=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 =C2=A0

c)=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 processing has stuck for a minute due to= some reason

=C2=A0=C2=A0 [Fri Aug 08 16:09:17 2014] [debug] ssl_engine_kernel.c(2118= ): [client 127.0.0.1] Certificate Verification, depth 0 [subject: /C=3Dy/ST= =3Dy/L=3Dy/O=3Dy/OU=3Dy/CN=3Dy, issuer: /C=3Dy/ST=3Dy/L=3Dy/O=3Dy/OU=3Dy/CN= =3Dy, serial: xyz]

=C2=A0=C2=A0 [Fri Aug 08 16:10:12 2014] [info] [client 10.150.90.25] Con= nection to child 6 established (server *:<port number from client>)

=C2=A0

d)=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 SSL handshake has started but didn=E2=80=99t c= omplete for 4 connections related to new process. There are no errors relat= ed to ssl

$ grep -i handshake errorlog.2014-08-08-07_0= 6_44 | grep -c start

707

$ grep -i handshake errorlog.2014-08-08-07_0= 6_44 | grep -c done

703

$

=C2=A0

Apache version:

2.2.15

=C2=A0

=C2=A0

Proxy setting:

SSLProxyEngine On

SSLProxyCipherSuite ALL

SSLProxyMachineCertificateFile /var/ssl/proxy.pem=

=C2=A0

proxyPass /app1 ht= tps://localhost:<port number>/app1 (Tomcat)

=C2=A0

Worker configuration:

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 15

<IfModule worker.c>

StartServers=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 1

MaxClients=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 25

MinSpareThreads=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 12<= /u>

MaxSpareThreads=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 25<= /u>

ThreadsPerChild=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 25<= /u>

ServerLimit=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 1

MaxRequestsPerChild=C2=A0 100

MaxMemFree=C2=A0 50

</IfModule>

=C2=A0

Thanks & Regards,

Krishna


MaxRequestsPerChild 100 is ridiculously low. =C2=A0What is happening in htt= pd to cause you to need that setting?

=C2=A0

Anyway...

=C2=A0

Once an httpd child process has reached 100 connecti= ons, it initiates a graceful shutdown, which means that instead of aborting= current requests it will instead wait for current requests to finish, then exit.

=C2=A0

During the time that it is waiting for current reque= sts to finish, new connections must be handled by other child processes. = =C2=A0BUT you set ServerLimit to 1 (and other directives such as ThreadsPerChild and MaxClients are consistent with allowing only o= ne child process), so no other child process can be created during that tim= e.

=C2=A0

Thus, once 100 connections are handled, new clients = will be blocked until existing requests finish.

=C2=A0

--/--

=C2=A0

My guess: =C2=A0Your Java application takes a long t= ime (maybe forever?) to handle some requests. =C2=A0MaxRequestsPerChild mak= es it worse. =C2=A0If the Java requests are slow and eventually finish, the solution is to keep a steady set of httpd child processes (having them= gracefully exit when there are slow backend requests can be harmful) and i= ncrease the number of httpd threads/child processes to handle the load.<= /u>

=C2=A0

If some Java requests hang, see how to handle that o= n the Tomcat side.


Enable server status with ExtendedStatus On and watc= h what happens -- whether or not certain requests handled by the Java appli= cation take a relatively long time, tieing up some or all of your very limited number of httpd threads.

=C2=A0

--

Born in Roswell... married an alien...
http://emptyhammock.= com/

=C2=A0

=C2=A0




--
Born in Roswell... married an alien...
http://emptyhammock.com/

--089e01493fa84eabed0501365687--