www-apache-bugdb mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Koh <chun...@pc.jaring.my>
Subject general/8303: Apache Self-Termination Problem
Date Thu, 06 Sep 2001 08:29:27 GMT

>Number:         8303
>Category:       general
>Synopsis:       Apache Self-Termination Problem
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    apache
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          support
>Submitter-Id:   apache
>Arrival-Date:   Thu Sep 06 01:30:00 PDT 2001
>Closed-Date:
>Last-Modified:
>Originator:     chunhoh@pc.jaring.my
>Release:        1.3.20
>Organization:
apache
>Environment:
Linux 2.2.16-3 #4 SMP Wed Dec 6 08:08:09 CST 2000 i686 unknown
libc-5.3.12-31
gcc version pgcc-2.95.2 19991024 (release)
Server version: Apache/1.3.20 (Unix)
>Description:
We are currently experiencing some strange Apache behavior at one of our Linux servers. We
would like to request for expertise in this area to assist us in this issue, and therefore,
we would greatly appreciate it if you can bring in an Apache consultant or expert as soon
as possible.

Below is a detailed description of the problem experienced and the measures that we have tried
to resolve the problem:

Since the night of September 4th, the web server was shutting down itself every 30 seconds
or 1 minute. As can be seen from the error log file /var/log/httpd/error_log,

[Tue Sep  4 21:56:56 2001] [notice] caught SIGTERM, shutting down
[Tue Sep  4 21:56:57 2001] [notice] Apache/1.3.12 (Unix) PHP/4.0.0 mod_ssl/2.6.6 OpenSSL/0.9.5a
configured -- resuming normal
 operations
[Tue Sep  4 21:56:57 2001] [notice] suEXEC mechanism enabled (wrapper: /etc/httpd/bin/suexec)
[Tue Sep  4 21:57:27 2001] [notice] caught SIGTERM, shutting down
[Tue Sep  4 21:57:28 2001] [notice] Apache/1.3.12 (Unix) PHP/4.0.0 mod_ssl/2.6.6 OpenSSL/0.9.5a
configured -- resuming normal
 operations
[Tue Sep  4 21:57:28 2001] [notice] suEXEC mechanism enabled (wrapper: /etc/httpd/bin/suexec)

I don't know where Apache is getting all those SIGTERM signals from. After it shuts itself
down, the httpd somehow manages to turn itself up again after around 25 seconds without any
human intervention.

We did not alter any httpd configuration before the problem occurred, so we believe it was
not caused by configuration error.

We suspected that apache was overloaded,  but then the number httpd processes was only around
200 - 300 at peak, of which shouldn't be a problem since we have other servers which can run
very well handling 300 - 400 httpd processes at any time.

I have looked up the Apache bug report archive and tried out a number of changes, such as:
- Upgraded apache from 1.3.12 to 1.3.20
- Tried alternative of Apache HTTPD 2.0
- Tried reducing the MaxClients to 150 and increasing to 500 in /etc/httpd/conf/httpd.conf
- Reduced number of VirtualHosts and log files in the /etc/httpd/conf/httpd.conf
- Ensured that the operating system's ulimit is set to unlimited
- Set maximum allowable TCP backlogs (set from sysctl) of operating system to 1200 (default
is usually 128)
- Rebooted server a number of times

Occassionally, I also received some warning messages such as:

[Thu Sep 06 03:12:04 2001] [warn] child process 2136 still did not exit, sending a SIGTERM
[Thu Sep 06 03:12:21 2001] [error] child process 1952 still did not exit, sending a SIGKILL
[Thu Sep 06 03:12:21 2001] [error] child process 2136 still did not exit, sending a SIGKILL
[Thu Sep 06 03:13:28 2001] [notice] caught SIGTERM, shutting down
[Thu Sep 06 03:14:35 2001] [crit] (98)Address already in use: make_sock: could not bind to
address
0.0.0.0:80
[Thu Sep 06 03:14:35 2001] [alert] no listening sockets available, shutting down

Does this indicate that Apache failed to terminate its childs, and tried sending SIGTERM to
them, but somehow also sent the signal to itself and terminated itself? Is it possible that
some library in the operating is corrupted, causing Apache problems in killing its own threads?

However, none of these steps resolved the problem. Moreover, there are no other errors from
Apache except the SIGTERM entries which keeps occuring in the error_log. Usually such problems
are caused by apache httpd.conf configuration error, but I can't locate the point of error
in this case. A second possiblity would be due to some operating system limitations, but of
which I do not know what and how to resolve.

In the Apache bug archive, some people have attributed similar problems to the old Redhat
or glibc, but our server currently runs Redhat 6.2 and glibc 2.1.3, both of which should be
sufficient.

Moreover, we have another server which is running an almost identical system and server configuration,
and is running fine there.

As a temporary measure, what I did was run a script that monitors the output at /var/log/httpd/error_log,
and immediately issues an apache restart whenver I detect the line "SIGTERM". This reduces
the downtime, but is still causing a lot of access problems.

Please advice us on this as we are completely stumped, and there are a number of mission-critical
services deployed on the system.



Yours Sincerely,
Andy Koh.
>How-To-Repeat:

>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted:
 [In order for any reply to be added to the PR database, you need]
 [to include <apbugs@Apache.Org> in the Cc line and make sure the]
 [subject line starts with the report component and number, with ]
 [or without any 'Re:' prefixes (such as "general/1098:" or      ]
 ["Re: general/1098:").  If the subject doesn't match this       ]
 [pattern, your message will be misfiled and ignored.  The       ]
 ["apbugs" address is not added to the Cc line of messages from  ]
 [the database automatically because of the potential for mail   ]
 [loops.  If you do not include this Cc, your reply may be ig-   ]
 [nored unless you are responding to an explicit request from a  ]
 [developer.  Reply only with text; DO NOT SEND ATTACHMENTS!     ]
 
 


Mime
View raw message