Mailing-List: contact users-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@httpd.apache.org
MIME-Version: 1.0
Date: Fri, 18 Dec 2015 18:30:01 +0100
Message-ID: 
 <CAN-a7TGJ6cDTObHcn1y26NQAuyh+En7_SQzT6Ca4W8_aJXDu4g@mail.gmail.com>
From: Rainer Canavan <rainer.canavan@sevenval.com>
To: users@httpd.apache.org
Content-Type: text/plain; charset=UTF-8
Subject: [users@httpd] AH00273: apr_proc_mutex_lock failed,
 possibly caused by cron, systemd
 or su

Hi,

we've got an obscure problem with the apache httpd that was shipped
with CentOS 7.2. We perform automatic builds and updates via cron,
and, since the update to CentOS 7.2. The update script is triggered by
cron and stops, yum updates and starts the httpd. When the next cron
job that is run as the same user as the httpd (not the update job)
terminates, the httpd frequently fails, starting with a AH00273:
apr_proc_mutex_lock failed message, and then a never ending loop of
AH00272 messages, one from each httpd process that is forked, until
the listener process is stopped.

[Thu Dec 17 08:30:04.895455 2015] [mpm_worker:notice] [pid 12021:tid
140011320178752] - AH00295: caught SIGTERM, shutting down
[Thu Dec 17 08:30:52.783949 2015] [mpm_worker:notice] [pid 23947:tid
140703866935360] - AH00292: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips
configured -- resuming normal operations
[Thu Dec 17 08:30:52.784091 2015] [core:notice] [pid 23947:tid
140703866935360] - AH00094: Command line: '/usr/sbin/httpd -d
/opt/sevenval/fit14/ -f /opt/sevenval/fit14/conf/httpd.conf'
[Thu Dec 17 08:40:54.959482 2015] [mpm_worker:emerg] [pid 23949:tid
140703388559104] (43)Identifier removed: - AH00273:
apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[Thu Dec 17 08:40:55.669532 2015] [mpm_worker:emerg] [pid 25505:tid
140703388559104] (22)Invalid argument: - AH00272: apr_proc_mutex_lock
failed before this child process served any requests.
[Thu Dec 17 08:40:55.991490 2015] [mpm_worker:emerg] [pid 23950:tid
140703388559104] (22)Invalid argument: - AH00273:
apr_proc_mutex_unlock failed. Attempting to shutdown process
gracefully.
[Thu Dec 17 08:40:57.672519 2015] [mpm_worker:emerg] [pid 25540:tid
140703388559104] (22)Invalid argument: - AH00272: apr_proc_mutex_lock
failed before this child process served any requests.
[Thu Dec 17 08:40:59.676573 2015] [mpm_worker:emerg] [pid 25575:tid
140703388559104] (22)Invalid argument: - AH00272: apr_proc_mutex_lock
failed before this child process served any requests.

The httpd version is the old version from RHEL (httpd-2.4.6-40.el7).
The backtrace is rather uninteresting:

(gdb) bt full
#0  accept_mutex_error (func=0x7f3fcac5d299 "unlock", rv=22,
process_slot=0) at worker.c:659
        level = 0
#1  0x00007f3fcac5b46a in listener_thread (thd=0x7f3fcdc405c8,
dummy=<optimized out>) at worker.c:849
        ti = <optimized out>
        process_slot = 0
        tpool = <optimized out>
        csd = 0x7f3fc00444c0
        ptrans = 0x7f3fc0044438
        pollset = 0x7f3fc003fa30
        rv = <optimized out>
        lr = 0x7f3fcdbd5a98
        have_idle_worker = 1
        last_poll_idx = 1
#2  0x00007f3fcc26cdc5 in start_thread (arg=0x7f3fb0fe1700) at
pthread_create.c:308
        __res = <optimized out>
        pd = 0x7f3fb0fe1700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139911529109248,
4832434510689815290, 0, 139911529109952, 139911529109248, 0,
-4796579440261772550,
                -4796375556103124230}, mask_was_saved = 0}}, priv =
{pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0,
canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#3  0x00007f3fcbd9621d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.

Does anyone habe any ideas what may cause these mutex errors?

thanks,

rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org