httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcin Wanat <marcin.wa...@gmail.com>
Subject [users@httpd] Apache httpd 2.4.4 many connections in Reading Request state
Date Wed, 27 Mar 2013 13:42:27 GMT
Hi,

i have weird problem with newly configured server.
I have running Apache 2.2.16(prefork) with php(module) 5.2.14 running
on CentOS5 for two years on my old server and last week i decided to
move to new server (CentOS6) and upgrade Apache to 2.4.4(prefork) and
php to 5.4.13.
On old server everything is working fine but when i route this traffic
(A DNS record of my domain) to new server then strange things are
happening.

For the first 30 seconds after httpd start everything is normal, and
then after 30 seconds about 25% of active connections are in "R"
(Reading Request) state. These connections are freezed  in R state for
10-25secs and then dissapear.
They are not logged in error_log even in trace6 level.

server-status for such connection looks like:
PID	Acc	M	CPU	SS	Req	Conn	Child	Slot	Client	VHost	
22182	0/90/90	R	0.73	3	5	0.0	0.03	0.03	81.96.86.224	

netstat for this IP shows:
tcp        0      0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52168   TIME_WAIT   -
tcp        0      0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52202   TIME_WAIT   -
tcp        0      0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52254   TIME_WAIT   -
tcp        0      0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52201   TIME_WAIT   -
tcp        0      0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52255   ESTABLISHED 22182/httpd
tcp        0      0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52169   TIME_WAIT   -
tcp        0      0 ::ffff:95.211.131.XXX:80
::ffff:81.96.86.224:52203   TIME_WAIT   -

Server load is about 0.5 and total CPU usage ~5%. Server is 16core,
32gb ram, 4xSAS hdd so it is not overloaded.
Network is 100mbps and usage of about 4mbps (only html content). I
have tested connection to few IPs which freeze in R state with mtr,
but no single packet loss were found.

Server is handling about 100requests/sec in peek hours and 50req/sec
during night.

Regardless of req/sec problem always occurs after about 30secs after
httpd start.
apachectl restart does not change anything. Only manually killing all
httpd processes and starting them from scratch can give 30seconds free
of R state connections.

I was also trying to recompile httpd with event mpm - effect is the
same. Even when i remove all htdocs content and every request is 404,
Reading Request connections appears after 30secs. Was trying httpd
2.2.24 and the effect was the same.

I was even trying different kernels (Centos6 2.6.32-358.2.1.el6 and
newest one 3.8.4) but with no luck.

These Reading Request connections are comming from different networks
all over the world.

I was also trying to do some stress tests with ab -n 50000 -c 20 from
different servers but each time these tests completed without any
freeze.

Some details:
[root@srv logs]# httpd -V
Server version: Apache/2.4.4 (Unix)
Server built:   Mar 24 2013 01:43:25
Server's Module Magic Number: 20120211:11
Server loaded:  APR 1.4.6, APR-UTIL 1.5.1
Compiled using: APR 1.4.6, APR-UTIL 1.5.1
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/usr/local/httpd"
 -D SUEXEC_BIN="/usr/local/httpd/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
[root@srv logs]# httpd -l
Compiled in modules:
  core.c
  mod_so.c
  http_core.c
  prefork.c

Loaded modules:
LoadModule php5_module        modules/libphp5.so
LoadModule unixd_module         modules/mod_unixd.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_core_module    modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule rewrite_module       modules/mod_rewrite.so
LoadModule mime_module       modules/mod_mime.so
LoadModule dir_module           modules/mod_dir.so
LoadModule authn_core_module    modules/mod_authn_core.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule status_module        modules/mod_status.so
LoadModule authz_user_module    modules/mod_authz_user.so
LoadModule auth_basic_module    modules/mod_auth_basic.so

Prefork config:
    StartServers             256
    MinSpareServers          256
    MaxSpareServers         512
    MaxRequestWorkers      1024
    MaxConnectionsPerChild   0



Currently i am out of ideas what else could i check to trace this problem.

Regards,
Marcin Wanat

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message