httpd-bugs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject [Bug 53555] Scoreboard full error with event/ssl
Date Mon, 05 Sep 2016 13:18:21 GMT
https://bz.apache.org/bugzilla/show_bug.cgi?id=53555

--- Comment #50 from Luca Toscano <toscano.luca@gmail.com> ---
Thanks a lot for the details Valentin, will try to add my thoughts inline:

(In reply to Valentin Gjorgjioski from comment #49)

> This started happening after recent upgrade of Ubuntu. Apache was the same,
> and now it is the same.  Ubuntu is 14.04.5 LTS, Apache is 2.4.7. 

This is a very old version of httpd, so if you could if would be really great
to upgrade Trusty to something more recent to see the differences.

> This is high load, production server. Working for 1.5 year without any
> problems so far. 
> 
> Here is some log of that update, when the problem started: 
> 
> [UPGRADE] apache2:amd64 2.4.7-1ubuntu4.9 -> 2.4.7-1ubuntu4.13
> [UPGRADE] apache2-bin:amd64 2.4.7-1ubuntu4.9 -> 2.4.7-1ubuntu4.13
> [UPGRADE] apache2-data:amd64 2.4.7-1ubuntu4.9 -> 2.4.7-1ubuntu4.13
> [UPGRADE] apache2-mpm-worker:amd64 2.4.7-1ubuntu4.9 -> 2.4.7-1ubuntu4.13
> [UPGRADE] apache2-utils:amd64 2.4.7-1ubuntu4.9 -> 2.4.7-1ubuntu4.13
> [INSTALL] php5-mysqlnd:amd64
> [UPGRADE] php5-cli:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-common:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-curl:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-fpm:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-gd:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-intl:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-pgsql:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-pspell:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-readline:amd64 5.5.9+dfsg-1ubuntu4.14 ->
> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-recode:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-sqlite:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-tidy:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-xmlrpc:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5-xsl:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> [UPGRADE] php5:amd64 5.5.9+dfsg-1ubuntu4.14 -> 5.5.9+dfsg-1ubuntu4.19
> 
> 
> Here is what I nailed it down to: 
> 1. After this upgrade I needed to DISABLE the opcache in PHP, because
> problems started with fatal errors and segmentation faults with wordpress. 
> 2. Because of the 1. the server got even higher load. 
> 3. Higher load caused full scoreboard, and maxRequestWorkersk.

Stating the obvious but the httpd issue seems to be a consequence of all the
php upgrades happened at the same time. Have you tried to rollback the last
upgrade to see if the issue persists?


>
> What I found were two problems: 
> 
> 1. When high load occurs and MaxReqeustWorkers is hit, the apache stops
> responding (dies). It should slow down, should not accept new requests until
> free slot, but it shouldn't stop responding.  I think I saw this reported
> somewhere else, e.g.: 
> https://www.digitalocean.com/community/questions/apache2-crash-on-ubuntu-14-
> 04-maxrequestworkers-issue

Would you mind to include the logs and/or more details about this? Again it
would be really great to know if the problem is the same with a more recent
version of httpd. 

> 
> 2. When I found a way to solve the problem with high load (enable wp cache
> plugins), now the second problem started, mainly on apache reload (log
> rotation) or even on regular basis WHEN MaxConnectionsPerChild is different
> from 0,  and/or when pm.max_requests is different from 0. Why this is a
> problem - because children are dying after certain numbers of requests, and
> then they get stuck into "G" state, and never completing. This is filling
> your scoreboard and you are ending with that error. Once you set these to 0,
> problem more or less disappears. 

Do you have long timeouts (proxy, etc..) in your httpd configuration? This
would be a useful information for us, it happened in the past that long proxy
timeouts where exacerbating the issue that you described.

> 
> Workaround is setting these to 0, and hoping all scripts are good, no memory
> leaks, lowering memory usage in php.ini, and restaring the server each day
> (on logrotate restart and not reload). 

> 
> Very important trick that I learned in during this is also this one: ALWAYS
> restart php-fpm and apache together. Failing to do so leads to some
> instabilities. 
> 
> For me that workaround work, but I would like to hear why this happens, and
> how we can prevent it (especially the problem when Apache dies when
> MaxRequestWorkers is readched).


As written above it would be great to know more about the "Apache dies" part.
Any detail that you could share with us would be really appreciated.

Thanks!

Luca

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


Mime
View raw message