Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9B2B3200C39 for ; Thu, 16 Mar 2017 18:05:12 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 99DD6160B78; Thu, 16 Mar 2017 17:05:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8A87A160B72 for ; Thu, 16 Mar 2017 18:05:11 +0100 (CET) Received: (qmail 57102 invoked by uid 500); 16 Mar 2017 17:05:10 -0000 Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: dev@httpd.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@httpd.apache.org Received: (qmail 57092 invoked by uid 99); 16 Mar 2017 17:05:10 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Mar 2017 17:05:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 0150DC3A94 for ; Thu, 16 Mar 2017 17:05:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id cEFbFRQvBM3M for ; Thu, 16 Mar 2017 17:05:04 +0000 (UTC) Received: from mail-wm0-f53.google.com (mail-wm0-f53.google.com [74.125.82.53]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 34AF85F238 for ; Thu, 16 Mar 2017 17:05:04 +0000 (UTC) Received: by mail-wm0-f53.google.com with SMTP id n11so53315688wma.0 for ; Thu, 16 Mar 2017 10:05:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=uolkMTIiqcbjr6DmcSLNNKDrjaU+3wvcC/kRkrC0ylc=; b=aoFOvDLLaU6KLpERWi/og23bwNC+q7PnLwCsh7N7nN4iYqD9BISvLQ2mkoJuM312jh x7cDoOjGJcaYS7Q282byJ6dTArTsc/2FM0otBSwAtFlL+08YJTAAN6lDE4DTcN10UuMh XXp3K6VxE+mDTajEAOAdtk5KWt4tz7ZB5XZ9A/cuX4sf6K+2D92/1v4uzK1vm4kQciyw F0Ul90+rL9q+DGwVP3dPeMccOSCuQdHvvn1JK3PG+sBmSQR01DF/P0pV0H2geBbJEVNg BfYXMrZCFSpvmneT3hNmF2zx4vOpYjwz5fts0SXqiqe+pXkeYLpowdCsl8PuUlO2fB4L G0CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=uolkMTIiqcbjr6DmcSLNNKDrjaU+3wvcC/kRkrC0ylc=; b=B7S9fUQfhsmd0n4zpWXfuDEQF+3TJ25V6SxzbQi8Nix5pZnTRJeAtJxnfUA4LJkmiW 6Nnwy3f7YbR/ngVeLaT73w/pfiB6B1MTEZ/lENwvOTbfeSEuTdwu0c7A36segFamqxU6 LBjlN+mpbIKjzdTYBqUxhGsXyl2pINqjYZSSg0t/EoX5ZmWXGqbrBwzT7il3hOYmARaV 0HqYI/yXLsVrc1ieFOzKzZuR2AC5QEGvD5m8Q2XsJvgQUzWVf8wdJgyv+Hz7pSNVnuZg slUz6JCu+aM0yu1BY9i/pT4enCZZ3UM/aTzlGUahRl6w7hKOE2RVWyuLtKCbMrVayZD0 OiPQ== X-Gm-Message-State: AFeK/H0Yd16OYxd2v0yk7RjmxbbIF3+fMuMxNxsRuWoK6G79w93nHGe0bBpe7TilvFollvIEsDNOvIm0XvVhzw== X-Received: by 10.28.130.212 with SMTP id e203mr9657131wmd.104.1489683903000; Thu, 16 Mar 2017 10:05:03 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.113.77 with HTTP; Thu, 16 Mar 2017 10:05:02 -0700 (PDT) In-Reply-To: <5699c080-c814-aae0-defa-b7d1ba1e5254@profihost.ag> References: <43c3a7ed-e86b-1543-a18f-255e747ad680@profihost.ag> <5699c080-c814-aae0-defa-b7d1ba1e5254@profihost.ag> From: Luca Toscano Date: Thu, 16 Mar 2017 18:05:02 +0100 Message-ID: Subject: Re: Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit To: Apache HTTP Server Development List Content-Type: multipart/alternative; boundary=001a11442c9eddb70d054adc0fdc archived-at: Thu, 16 Mar 2017 17:05:12 -0000 --001a11442c9eddb70d054adc0fdc Content-Type: text/plain; charset=UTF-8 2017-03-16 15:24 GMT+01:00 Stefan Priebe - Profihost AG < s.priebe@profihost.ag>: > Hi, > > Am 16.03.2017 um 12:26 schrieb Luca Toscano: > > Hi Stefan, > > > > 2017-03-16 12:14 GMT+01:00 Stefan Priebe - Profihost AG > > >: > > > > Hi Yann, > > > > no sure whether this is due to your mpm event patch. > > > > From time to time i see the following error mesages in my logs and > the > > only chance to fix it is to restart apache. > > > > [Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid > > 140212799559552] AH: scoreboard is full, not at > > MaxRequestWorkers.Increase > > ServerLimit. > > [Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid > > 140212799559552] AH: scoreboard is full, not at > > MaxRequestWorkers.Increase > > ServerLimit. > > [Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid > > 140212799559552] AH: scoreboard is full, not at > > MaxRequestWorkers.Increase > > ServerLimit. > > [Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid > > 140212799559552] AH: scoreboard is full, not at > > MaxRequestWorkers.Increase > > ServerLimit. > > [Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid > > 140212799559552] AH: scoreboard is full, not at > > MaxRequestWorkers.Increase > > ServerLimit. > > [Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid > > 140212799559552] AH: scoreboard is full, not at > > MaxRequestWorkers.Increase > > ServerLimit. > > > > Settings: > > ServerLimit 50 > > ThreadLimit 200 > > ThreadsPerChild 200 > > MinSpareThreads 200 > > MaxSpareThreads 400 > > MaxClients 10000 > > MaxRequestsPerChild 10000 > > > > MaxRequestWorkers isn't set at all. > > > > > > I believe that MaxClients (its old name) takes the place of > > MaxRequestWorkers, but it is set as the default value (ServerLimit * > > ThreadsPerChild). From 2.4.25 onwards mpm-event offers a new > > functionality to handle Gracefully terminating processes > > (https://httpd.apache.org/docs/current/mod/event.html#how-it-works), but > > it must be tuned raising the value of ServerLimit (more details in the > > docs). > > Hui didn't know this. So i should remove ALL settings and only set: > ThreadsPerChild > ServerLimit > MaxRequestWorkers > AsyncRequestWorkerFactor > > ? is this true? I'm missing some examples also considering MinSpare and > ThreadLimit or are they no longer needed? > I would simply replace MaxClients with MaxRequestWorkers keeping the rest of your config (that it is still valid, all the settings that you mentioned are still used by mpm-event), without touching AsyncRequestWorkerFactor (unless you want to play with it but the default is generally good). About your specific max scoreboard issue, I would: 1) try to raise ServerLimit to allow more space for slots occupied by processes in G state (graceful termination), as indicated in the docs. 2) Follow Daniel's suggestion about Max/Min spare workers. > > This is only a speculation from my side, to have a better idea of what's > > happening it would be great to see how the Scoreboard looks like in > > server status, and if the error status happens during specific events > > like graceful reload for log rotation. > > All of them happened after a reload - but I'll recheck. I'm pretty sure > that the /server-status page was no longer responding. Is there any > other way to get the status of of httpd while it does no longer serve > pages? I am not aware of any other way sadly, but hopefully you will not need it with the new settings :) Do you have long running http connections that can keep httpd processes in the G state after reload? Usually this is the main problem, and very easy to test. Let us know how it goes! Luca --001a11442c9eddb70d054adc0fdc Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


2017-03-16 15:24 GMT+01:00 Stefan Priebe - Profihost AG <s.priebe@= profihost.ag>:
Hi,

Am 16.03.2017 um 12:26 schrieb Luca Toscano:
> Hi Stefan,
>
> 2017-03-16 12:14 GMT+01:00 Stefan Priebe - Profihost AG
> <s.priebe@profihost= .ag <mailto:s.priebe@profih= ost.ag>>:
>
>=C2=A0 =C2=A0 =C2=A0Hi Yann,
>
>=C2=A0 =C2=A0 =C2=A0no sure whether this is due to your mpm event patch= .
>
>=C2=A0 =C2=A0 =C2=A0From time to time i see the following error mesages= in my logs and the
>=C2=A0 =C2=A0 =C2=A0only chance to fix it is to restart apache.
>
>=C2=A0 =C2=A0 =C2=A0[Thu Mar 16 01:00:35.445184 2017] [mpm_event:error]= [pid 27485:tid
>=C2=A0 =C2=A0 =C2=A0140212799559552] AH: scoreboard is full, not at
>=C2=A0 =C2=A0 =C2=A0MaxRequestWorkers.Increase
>=C2=A0 =C2=A0 =C2=A0ServerLimit.
>=C2=A0 =C2=A0 =C2=A0[Thu Mar 16 01:00:36.446178 2017] [mpm_event:error]= [pid 27485:tid
>=C2=A0 =C2=A0 =C2=A0140212799559552] AH: scoreboard is full, not at
>=C2=A0 =C2=A0 =C2=A0MaxRequestWorkers.Increase
>=C2=A0 =C2=A0 =C2=A0ServerLimit.
>=C2=A0 =C2=A0 =C2=A0[Thu Mar 16 01:00:37.447181 2017] [mpm_event:error]= [pid 27485:tid
>=C2=A0 =C2=A0 =C2=A0140212799559552] AH: scoreboard is full, not at
>=C2=A0 =C2=A0 =C2=A0MaxRequestWorkers.Increase
>=C2=A0 =C2=A0 =C2=A0ServerLimit.
>=C2=A0 =C2=A0 =C2=A0[Thu Mar 16 01:00:38.448177 2017] [mpm_event:error]= [pid 27485:tid
>=C2=A0 =C2=A0 =C2=A0140212799559552] AH: scoreboard is full, not at
>=C2=A0 =C2=A0 =C2=A0MaxRequestWorkers.Increase
>=C2=A0 =C2=A0 =C2=A0ServerLimit.
>=C2=A0 =C2=A0 =C2=A0[Thu Mar 16 01:00:39.449185 2017] [mpm_event:error]= [pid 27485:tid
>=C2=A0 =C2=A0 =C2=A0140212799559552] AH: scoreboard is full, not at
>=C2=A0 =C2=A0 =C2=A0MaxRequestWorkers.Increase
>=C2=A0 =C2=A0 =C2=A0ServerLimit.
>=C2=A0 =C2=A0 =C2=A0[Thu Mar 16 01:00:40.450184 2017] [mpm_event:error]= [pid 27485:tid
>=C2=A0 =C2=A0 =C2=A0140212799559552] AH: scoreboard is full, not at
>=C2=A0 =C2=A0 =C2=A0MaxRequestWorkers.Increase
>=C2=A0 =C2=A0 =C2=A0ServerLimit.
>
>=C2=A0 =C2=A0 =C2=A0Settings:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ServerLimit=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A050
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ThreadLimit=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0200
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ThreadsPerChild=C2=A0 = =C2=A0 =C2=A0 =C2=A0 200
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0MinSpareThreads=C2=A0 = =C2=A0 =C2=A0 =C2=A0 200
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0MaxSpareThreads=C2=A0 = =C2=A0 =C2=A0 =C2=A0 400
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0MaxClients=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A010000
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0MaxRequestsPerChild=C2= =A0 =C2=A0 10000
>
>=C2=A0 =C2=A0 =C2=A0MaxRequestWorkers isn't set at all.
>
>
> I believe that MaxClients (its old name) takes the place of
> MaxRequestWorkers, but it is set as the default value (ServerLimit * > ThreadsPerChild). From 2.4.25 onwards mpm-event offers a new
> functionality to handle Gracefully terminating processes
> (https://httpd.apache.org/docs/current/mod/event.html#how-it-works), but
> it must be tuned raising the value of ServerLimit (more details in the=
> docs).

Hui didn't know this. So i should remove ALL settings and o= nly set:
ThreadsPerChild
ServerLimit
MaxRequestWorkers
AsyncRequestWorkerFactor

? is this true? I'm missing some examples also considering MinSpare and=
ThreadLimit or are they no longer needed?

I would simply replace MaxClients with MaxRequestWorkers keeping the res= t of your config (that it is still valid, all the settings that you mention= ed are still used by mpm-event), without touching =C2=A0AsyncRequestWorkerF= actor (unless you want to play with it but the default is generally good). = About your specific max scoreboard issue, I would:

1) try to raise ServerLimit to allow more space for slots occupied by proc= esses in G state (graceful termination), as indicated in the docs.
2) Follow Daniel's suggestion about Max/Min spare workers.
=
=C2=A0
> This is only a speculation from my side, to have a better idea of what= 's
> happening it would be great to see how the Scoreboard looks like in > server status, and if the error status happens during specific events<= br> > like graceful reload for log rotation.

All of them happened after a reload - but I'll recheck. I'm = pretty sure
that the /server-status page was no longer responding. Is there any
other way to get the status of of httpd while it does no longer serve pages= ?

I am not aware of any other way sadly, bu= t hopefully you will not need it with the new settings :)

Do you have long running http connections that can keep httpd proce= sses in the G state after reload? Usually this is the main problem, and ver= y easy to test.

Let us know how it goes!
=

Luca=C2=A0

--001a11442c9eddb70d054adc0fdc--