httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nourredine Rouibah <nourredine.roui...@gmail.com>
Subject [users@httpd] Apache httpd 2.2.12 hangs
Date Tue, 20 Aug 2013 08:03:39 GMT
Hello,

My apologizes, my post of yesterday included HTML tables from server-status
output.(cut/paste).
Here is a more digest version.

Our httpd Apache server keeps hanging every day or many times a day
depending on the activity.
Apache is the frontend of our Subversion 1.6.5 server.
We have about a 100 client developers accessing SVN thru Apache and a few
Hudson continuous integration servers accessing SVN as well and running
many automatic SVN accesses 24/7.
SVN is working well because when the problem happens, I can isssue local
SVN commands outside of httpd without any problem.

Our Apache is listening to 2 different ports 20000 and 20001.
Port 20000 is the standard SVN port and port 20001 is used by the SVN PHP
Admin tools. (svnadmin and websvn)

When the httpd Apache hangs it is always on port 20000.
All SVN requests are stuck for some reasons.

When the httpd hangs on port 20000, if we issue an http request using
svnadmin on port 20001 then the server is unblocked for a short moment
sometime few minutes after hanging again.
Then if we re-issue a request on port 20001 it will again release port
20000 requests for a while.
The only clean solution is to restart the httpd server.
Then it will eventualy run for a few hours or a day or two and then hang
again.

We turned keepalive to OFF to test if we were hiting socket release issues
but the problem just happened with KEEPALIVE set to OFF.
Our server is a large solaris machine with plenty of memory and swap.
Solaris : SunOS MyMachine 5.10 Generic_147440-24 sun4v sparc sun4v
Apache: Server Version: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d
DAV/2 SVN/1.6.5 PHP/5.3.1
------------------------------------------------------------------------
Here is the extract from our httpd.conf file

ServerRoot "/opt/xxxxxx/xxx/xxxxxxxxxxxxxxxxxxxxxxx"
LockFile accept.lock
PidFile apache2.pid

Listen 0.0.0.0:20001
LockFile accept.lock
PidFile apache2.pid
Timeout 300
KeepAlive Off # was On before
MaxKeepAliveRequests 1536
KeepAliveTimeout 2

<IfModule mpm_prefork_module>
  StartServers          32
  MinSpareServers       48
  MaxSpareServers       64
  ServerLimit          512
  MaxClients            96
  MaxRequestsPerChild 1536
</IfModule>
[...]
<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/opt/blablabla/cgi-bin/"
</IfModule>

<Directory "/opt/blablabla/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

# Main server Admin SVN
ServerName scm.xyz.com
DocumentRoot "/opt/blablabla/htdocs"

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

<Directory "/opt/blablabla/htdocs">
    Options Indexes FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

# PHP5 :
Include conf/extra/httpd-php.conf

# Repository SVN
Listen 0.0.0.0:20000
<VirtualHost *:20000>
    Include conf/extra/httpd-svn.conf
</VirtualHost>
------------------------------------------------------------------------
After the apache is hung here is the status-server request:

Apache Server Status for scm.xyz.com
Server Version: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d DAV/2
SVN/1.6.5 PHP/5.3.1
Server Built: Jan 13 2010 15:49:24
Current Time: Friday, 16-Aug-2013 15:21:52 CEST
Restart Time: Friday, 16-Aug-2013 03:10:02 CEST
Parent Server Generation: 0
Server uptime: 12 hours 11 minutes 50 seconds

Total accesses: 228561 - Total Traffic: 369.5 MB
CPU Usage: u637.01 s183.88 cu1.4 cs0 - 1.87% CPU load
5.21 requests/sec - 8.6 kB/second - 1694 B/request
8 requests currently being processed, 55 idle workers

_______________________________WWW_W___W______WWR______________.
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................

Srv PID Acc M CPU SS Req Conn       Child       Slot       Client
VHost       Request

0-0 20113  0/751/3794  _  7.48  606  6  0.0  0.49  2.95  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
1-0  27522  0/640/3674  _  12.28  627  7  0.0  0.49  3.04  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1
2-0  20194  0/735/3776  _  7.80  627  6  0.0  0.42  4.36  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
3-0 27523   0/647/3685  _  44.08  442  162540  0.0  20.14  21.82
xx.yyy.yy.108  myserver.a.b.c  REPORT /my.path/!svn/blablabla/default
HTTP/1.1
4-0 3076  0/1391/3425  W  14.89  0  0  0.0  0.90  2.42  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk HTTP/1.
5-0 10815  0/397/3735  _  4.42  627  18  0.0  0.24  3.12  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/bblablabla/trunk H
6-0 27524  0/643/3682  _  13.14  606  33  0.0  1.68  3.58  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/vcc/default HTTP/1.1
7-0 28908  0/633/3666  _  7.06  627  9  0.0  0.42  5.62  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/vcc/default HTTP/1.1
8-0 27525  0/640/3673  _  13.35  627  9  0.0  4.97  7.44  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
9-0 27527  0/639/3675  _  27.84  627  11  0.0  2.62  5.38  xx.yyy.yy.114
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
10-0 27528  0/638/3677  _  8.09  606  3  0.0  0.60  7.14  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla HTTP/1.1
11-0  16355  0/301/3469  _  3.08  627  6  0.0  0.18  4.83  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla HTTP/1.1
12-0  2761  0/989/3282  _  10.41  627  6  0.0  0.67  2.72  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
13-0  29771  0/632/3667  _  6.51  627  16  0.0  0.38  3.40  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/trunk HT
14-0  29928  0/586/3621  _  8.50  627  5  0.0  0.49  10.14  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla HTTP/1.1
15-0  29772  0/629/3666  _  6.77  606  8  0.0  0.37  3.15  xx.yyy.yy.108
myserver.a.b.c  REPORT /my.path/!svn/blablabla HTTP/1.1
16-0  27529  0/642/3682  _  6.56  627  6  0.0  0.36  3.67  xx.yyy.yy.114
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk HTTP/1.1
17-0  27530  0/631/3672  _  11.19  606  6  0.0  0.43  3.80  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk HTTP/1.1
18-0  27827  0/641/3678  _  6.79  615  5  0.0  0.38  2.20  xx.yyy.yy.140
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
19-0  28230  0/640/3098  _  7.22  627  6  0.0  0.39  1.72  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
20-0  28302  0/639/3679  _  7.11  606  6  0.0  0.48  4.59  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
21-0  28231  0/639/3675  W  8.22  0  0  0.0  0.60  4.22  xx.yyy.yy.5
myserver.a.b.c  GET /my.path/ HTTP/1.1
22-0  3077  0/1373/2899  _  26.72  627  17  0.0  3.16  10.28
xx.yyy.yy.108  myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/trunk
23-0  20679  0/235/3408  _  2.49  606  15  0.0  0.13  2.20  xx.yyy.yy.108
myserver.a.b.c  REPORT /my.path/!svn/blablabla/default HTTP/1.1
24-0  3078  0/1386/2905  _  16.83  627  14  0.0  2.16  3.08  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablablat/trunk
25-0  28297  0/616/2131  _  7.94  627  11  0.0  0.59  11.05  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
26-0  28299  0/634/3674  _  8.10  627  15  0.0  0.53  4.19  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/trunk
27-0  28300  0/641/2157  _  6.58  606  5  0.0  0.34  1.12  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla HTTP/1.1
28-0  28303  0/642/2159  _  6.82  615  5  0.0  0.43  1.61  xx.yyy.yy.140
myserver.a.b.c  PROPFIND /my.path HTTP/1.1
29-0  28304  0/639/3673  _  7.29  615  10  0.0  0.50  7.08  xx.yyy.yy.140
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla HTTP/1.1
30-0  28305  0/633/2149  W  7.62  0  0  0.0  0.44  3.47  xx.yyy.yy.119
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
31-0  29929  0/585/3619  W  6.31  0  0  0.0  0.41  4.00  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk HTTP/1.1
32-0  29930  0/592/3627  W  7.45  0  0  0.0  0.55  2.51  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk HTTP/1.
33-0  28910  0/635/3673  W  8.87  0  0  0.0  0.38  2.44  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk HTTP/1.1
34-0  28696  0/641/3676  _  6.57  606  98  0.0  0.38  10.64  xx.yyy.yy.108
myserver.a.b.c  REPORT /my.path/!svn/blablabla/default HTTP/1.1
35-0  6557  0/481/3517  W  4.98  0  0  0.0  0.33  8.89  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk H
36-0  28700  0/633/3671  _  7.61  627  6  0.0  0.38  2.22  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
37-0  29931  0/588/3621  _  6.22  627  14  0.0  0.36  13.57  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/testsSecurity
38-0  28697  0/626/3661  _  9.76  606  71  0.0  0.86  3.63  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
39-0  29766  0/628/3662  W  6.74  0  0  0.0  0.42  4.28  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/trunk/ HTTP/1.1
40-0  29932  0/586/3622  _  7.25  606  4  0.0  0.40  2.32  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
41-0  13813  0/366/3616  _  3.58  606  6  0.0  0.22  5.97  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
42-0  9  0/570/3606  _  5.83  627  13  0.0  0.37  21.85  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla HTTP/1.1
43-0  29770  0/634/3668  _  7.88  627  11  0.0  0.51  4.62  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
44-0  29916  0/614/3651  _  8.82  627  16  0.0  2.77  4.50  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
45-0  29905  0/631/3670  _  6.87  627  8  0.0  0.37  6.40  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
46-0  29906  0/623/3658  W  7.33  0  0  0.0  0.42  5.68  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk HTTP/1
47-0  29907  0/627/2494  W  7.11  0  0  0.0  0.51  1.57  xx.yyy.yy.243
myserver.a.b.c  GET /server-status HTTP/1.1
48-0  29917  0/615/3648  W  6.73  0  0  0.0  1.35  4.79  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk HTTP/1
49-0  29918  0/607/3640  _  9.02  606  6  0.0  0.79  3.13  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
50-0  29919  0/614/3648  _  6.30  627  17  0.0  0.35  2.08  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/trunk HTTP/
51-0  29933  0/587/3620  W  8.42  0  0  0.0  0.41  2.94  xx.yyy.yy.5
myserver.a.b.c  GET /my.path/trunk/ HTTP/1.1
52-0  29934  0/583/3618  _  7.54  606  6  0.0  0.47  3.41  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /blablabla/trunk HTTP/1.1
53-0  29935  0/593/3630  _  7.32  615  5  0.0  0.64  3.50  xx.yyy.yy.140
myserver.a.b.c  PROPFIND blablabla HTTP/1.1
54-0  6566  0/467/3504  W  5.79  0  0  0.0  0.28  2.77  xx.yyy.yy.140
myserver.a.b.c  PROPFIND /blablabla HTTP/1.1
55-0  6567  0/473/3508  _  4.58  606  6  0.0  0.27  2.09  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
56-0  27600  0/132/1651  _  1.71  627  9  0.0  0.14  7.04  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
57-0  27616  0/63/1602  _  29.34  615  5  0.0  19.80  22.77  xx.yyy.yy.140
myserver.a.b.c  PROPFIND /my.path HTTP/1.1
58-0  27617  0/118/1288  _  1.65  627  6  0.0  0.08  0.86  xx.yyy.yy.114
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1.1
59-0  27618  0/101/1277  W  0.96  0  0  0.0  0.07  3.19  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk HTTP/1.1
60-0  27619  0/101/3137  _  1.22  627  5  0.0  0.08  2.15  xx.yyy.yy.114
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
61-0  27621  0/101/3133  _  1.01  615  5  0.0  0.06  1.77  xx.yyy.yy.140
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
62-0  27623  0/99/3133  _  1.44  627  18  0.0  0.09  1.81  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla
63-0      -  0/0/1513  .  19.93  10227  11  0.0  0.00  1.34  xx.yyy.yy.108
myserver.a.b.c  OPTIONS /my.path/blablabla/trunk HTTP/1.1
64-0      -  0/0/1513  .  16.68  10235  6  0.0  0.00  1.06  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
65-0      -  0/0/1511  .  51.59  10167  9  0.0  0.00  20.72  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/4389 HTTP/1.1
66-0      -  0/0/1520  .  17.51  4947  7  0.0  0.00  1.19  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/blablabla/trunk HTTP/1
67-0      -  0/0/1520  .  17.32  4998  7  0.0  0.00  1.32  xx.yyy.yy.30
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
68-0      -  0/0/1520  .  29.50  4995  10  0.0  0.00  2.92  xx.yyy.yy.30
myserver.a.b.c  REPORT /my.path/!svn/blablabla/branches HTTP/1.1
69-0      -  0/0/784  .  8.89  12044  0  0.0  0.00  0.50  127.0.0.1
myserver.a.b.c  OPTIONS * HTTP/1.0
70-0      -  0/0/1524  .  17.60  4849  46  0.0  0.00  2.01  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
71-0      -  0/0/1520  .  28.70  4850  18  0.0  0.00  2.42  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/
72-0      -  0/0/1520  .  15.69  5015  7  0.0  0.00  0.89  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
73-0      -  0/0/1520  .  31.96  4955  19  0.0  0.00  8.35  xx.yyy.yy.108
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla
74-0      -  0/0/1521  .  18.82  4998  4  0.0  0.00  0.95  xx.yyy.yy.30
myserver.a.b.c  PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
75-0      -  0/0/1520  .  16.71  5015  8  0.0  0.00  0.96  xx.yyy.yy.108
myserver.a.b.c    PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
------------------------------------------------------------------------

Any help appreciated.

/Nourrédine

Mime
View raw message