httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Eissing <stefan.eiss...@greenbytes.de>
Subject Re: Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit
Date Fri, 24 Mar 2017 10:28:18 GMT
Stefan,

if you consider configuration changes, you could also set the H2 worker threads to a constant
number. They have a minimum and maximum where mod_http2 creates new worker when needed and
releases some when idle. If you have not specified anything via H2MinWorker/H2MaxWorker, then
it takes defaults from the MPM configuration.

I think the code for adding/removing workers is not totally safe. Can you set both configs
to the same number, so it will not trigger? Sth like

  H2MinWorkers 25
  H2MaxWorkers 25

(with a value >= ThreadsPerChild, any value high enough to serve your connections will
do)

I would like to know if that influences the spurious segfaults you see. Probably not, but
seems easy to verify.

Thanks!

-Stefan

> Am 16.03.2017 um 20:47 schrieb Stefan Priebe - Profihost AG <s.priebe@profihost.ag>:
> 
> Hi,
> 
> Am 16.03.2017 um 18:05 schrieb Luca Toscano:
>> 
>> 
>> 2017-03-16 15:24 GMT+01:00 Stefan Priebe - Profihost AG
>> <s.priebe@profihost.ag <mailto:s.priebe@profihost.ag>>:
>> 
>>    Hi,
>> 
>>    Am 16.03.2017 um 12:26 schrieb Luca Toscano:
>>> Hi Stefan,
>>> 
>>> 2017-03-16 12:14 GMT+01:00 Stefan Priebe - Profihost AG
>>> <s.priebe@profihost.ag <mailto:s.priebe@profihost.ag>
>>    <mailto:s.priebe@profihost.ag <mailto:s.priebe@profihost.ag>>>:
>>> 
>>>    Hi Yann,
>>> 
>>>    no sure whether this is due to your mpm event patch.
>>> 
>>>    From time to time i see the following error mesages in my logs
>>    and the
>>>    only chance to fix it is to restart apache.
>>> 
>>>    [Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid
>>>    140212799559552] AH: scoreboard is full, not at
>>>    MaxRequestWorkers.Increase
>>>    ServerLimit.
>>>    [Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid
>>>    140212799559552] AH: scoreboard is full, not at
>>>    MaxRequestWorkers.Increase
>>>    ServerLimit.
>>>    [Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid
>>>    140212799559552] AH: scoreboard is full, not at
>>>    MaxRequestWorkers.Increase
>>>    ServerLimit.
>>>    [Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid
>>>    140212799559552] AH: scoreboard is full, not at
>>>    MaxRequestWorkers.Increase
>>>    ServerLimit.
>>>    [Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid
>>>    140212799559552] AH: scoreboard is full, not at
>>>    MaxRequestWorkers.Increase
>>>    ServerLimit.
>>>    [Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid
>>>    140212799559552] AH: scoreboard is full, not at
>>>    MaxRequestWorkers.Increase
>>>    ServerLimit.
>>> 
>>>    Settings:
>>>            ServerLimit           50
>>>            ThreadLimit           200
>>>            ThreadsPerChild        200
>>>            MinSpareThreads        200
>>>            MaxSpareThreads        400
>>>            MaxClients             10000
>>>            MaxRequestsPerChild    10000
>>> 
>>>    MaxRequestWorkers isn't set at all.
>>> 
>>> 
>>> I believe that MaxClients (its old name) takes the place of
>>> MaxRequestWorkers, but it is set as the default value (ServerLimit *
>>> ThreadsPerChild). From 2.4.25 onwards mpm-event offers a new
>>> functionality to handle Gracefully terminating processes
>>> (https://httpd.apache.org/docs/current/mod/event.html#how-it-works
>>    <https://httpd.apache.org/docs/current/mod/event.html#how-it-works>), but
>>> it must be tuned raising the value of ServerLimit (more details in the
>>> docs).
>> 
>>    Hui didn't know this. So i should remove ALL settings and only set:
>>    ThreadsPerChild
>>    ServerLimit
>>    MaxRequestWorkers
>>    AsyncRequestWorkerFactor
>> 
>>    ? is this true? I'm missing some examples also considering MinSpare and
>>    ThreadLimit or are they no longer needed?
>> 
>> 
>> I would simply replace MaxClients with MaxRequestWorkers keeping the
>> rest of your config (that it is still valid, all the settings that you
>> mentioned are still used by mpm-event), without touching
>> AsyncRequestWorkerFactor (unless you want to play with it but the
>> default is generally good). About your specific max scoreboard issue, I
>> would:
>> 
>> 1) try to raise ServerLimit to allow more space for slots occupied by
>> processes in G state (graceful termination), as indicated in the docs.
>> 2) Follow Daniel's suggestion about Max/Min spare workers.
> 
> Will try to fix this. Need to implement this in our env. I'll most
> probably need until Monday.
> 
> 
>>> This is only a speculation from my side, to have a better idea of what's
>>> happening it would be great to see how the Scoreboard looks like in
>>> server status, and if the error status happens during specific events
>>> like graceful reload for log rotation.
>> 
>>    All of them happened after a reload - but I'll recheck. I'm pretty sure
>>    that the /server-status page was no longer responding. Is there any
>>    other way to get the status of of httpd while it does no longer
>>    serve pages?
>> 
>> 
>> I am not aware of any other way sadly, but hopefully you will not need
>> it with the new settings :)
>> 
>> Do you have long running http connections that can keep httpd processes
>> in the G state after reload? Usually this is the main problem, and very
>> easy to test.
>> 
>> Let us know how it goes!
>> 
>> Luca 

Stefan Eissing

<green/>bytes GmbH
Hafenstrasse 16
48155 M√ľnster
www.greenbytes.de


Mime
View raw message