httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Trawick <traw...@gmail.com>
Subject Re: [users@httpd] Apache httpd 2.4.4 many connections in Reading Request state
Date Wed, 27 Mar 2013 20:22:50 GMT
On Wed, Mar 27, 2013 at 9:42 AM, Marcin Wanat <marcin.wanat@gmail.com> wrote:
> Hi,
>
> i have weird problem with newly configured server.
> I have running Apache 2.2.16(prefork) with php(module) 5.2.14 running
> on CentOS5 for two years on my old server and last week i decided to
> move to new server (CentOS6) and upgrade Apache to 2.4.4(prefork) and
> php to 5.4.13.
> On old server everything is working fine but when i route this traffic
> (A DNS record of my domain) to new server then strange things are
> happening.
>
> For the first 30 seconds after httpd start everything is normal, and
> then after 30 seconds about 25% of active connections are in "R"
> (Reading Request) state. These connections are freezed  in R state for
> 10-25secs and then dissapear.
> They are not logged in error_log even in trace6 level.
>
> server-status for such connection looks like:
> PID     Acc     M       CPU     SS      Req     Conn    Child   Slot    Client  VHost
> 22182   0/90/90 R       0.73    3       5       0.0     0.03    0.03    81.96.86.224
>
> netstat for this IP shows:
> tcp        0      0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52168   TIME_WAIT   -
> tcp        0      0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52202   TIME_WAIT   -
> tcp        0      0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52254   TIME_WAIT   -
> tcp        0      0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52201   TIME_WAIT   -
> tcp        0      0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52255   ESTABLISHED 22182/httpd
> tcp        0      0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52169   TIME_WAIT   -
> tcp        0      0 ::ffff:95.211.131.XXX:80
> ::ffff:81.96.86.224:52203   TIME_WAIT   -
>
> Server load is about 0.5 and total CPU usage ~5%. Server is 16core,
> 32gb ram, 4xSAS hdd so it is not overloaded.
> Network is 100mbps and usage of about 4mbps (only html content). I
> have tested connection to few IPs which freeze in R state with mtr,
> but no single packet loss were found.
>
> Server is handling about 100requests/sec in peek hours and 50req/sec
> during night.
>
> Regardless of req/sec problem always occurs after about 30secs after
> httpd start.
> apachectl restart does not change anything. Only manually killing all
> httpd processes and starting them from scratch can give 30seconds free
> of R state connections.
>
> I was also trying to recompile httpd with event mpm - effect is the
> same. Even when i remove all htdocs content and every request is 404,
> Reading Request connections appears after 30secs. Was trying httpd
> 2.2.24 and the effect was the same.
>
> I was even trying different kernels (Centos6 2.6.32-358.2.1.el6 and
> newest one 3.8.4) but with no luck.
>
> These Reading Request connections are comming from different networks
> all over the world.
>
> I was also trying to do some stress tests with ab -n 50000 -c 20 from
> different servers but each time these tests completed without any
> freeze.
>
> Some details:
> [root@srv logs]# httpd -V
> Server version: Apache/2.4.4 (Unix)
> Server built:   Mar 24 2013 01:43:25
> Server's Module Magic Number: 20120211:11
> Server loaded:  APR 1.4.6, APR-UTIL 1.5.1
> Compiled using: APR 1.4.6, APR-UTIL 1.5.1
> Architecture:   64-bit
> Server MPM:     prefork
>   threaded:     no
>     forked:     yes (variable process count)
> Server compiled with....
>  -D APR_HAS_SENDFILE
>  -D APR_HAS_MMAP
>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>  -D APR_USE_SYSVSEM_SERIALIZE
>  -D APR_USE_PTHREAD_SERIALIZE
>  -D APR_HAS_OTHER_CHILD
>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>  -D DYNAMIC_MODULE_LIMIT=256
>  -D HTTPD_ROOT="/usr/local/httpd"
>  -D SUEXEC_BIN="/usr/local/httpd/bin/suexec"
>  -D DEFAULT_PIDLOG="logs/httpd.pid"
>  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>  -D DEFAULT_ERRORLOG="logs/error_log"
>  -D AP_TYPES_CONFIG_FILE="conf/mime.types"
>  -D SERVER_CONFIG_FILE="conf/httpd.conf"
> [root@srv logs]# httpd -l
> Compiled in modules:
>   core.c
>   mod_so.c
>   http_core.c
>   prefork.c
>
> Loaded modules:
> LoadModule php5_module        modules/libphp5.so
> LoadModule unixd_module         modules/mod_unixd.so
> LoadModule authz_host_module modules/mod_authz_host.so
> LoadModule authz_core_module    modules/mod_authz_core.so
> LoadModule access_compat_module modules/mod_access_compat.so
> LoadModule rewrite_module       modules/mod_rewrite.so
> LoadModule mime_module       modules/mod_mime.so
> LoadModule dir_module           modules/mod_dir.so
> LoadModule authn_core_module    modules/mod_authn_core.so
> LoadModule authn_file_module modules/mod_authn_file.so
> LoadModule status_module        modules/mod_status.so
> LoadModule authz_user_module    modules/mod_authz_user.so
> LoadModule auth_basic_module    modules/mod_auth_basic.so
>
> Prefork config:
>     StartServers             256
>     MinSpareServers          256
>     MaxSpareServers         512
>     MaxRequestWorkers      1024
>     MaxConnectionsPerChild   0

What are the equivalent settings in your 2.2 setup?

>
>
>
> Currently i am out of ideas what else could i check to trace this problem.
>
> Regards,
> Marcin Wanat
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>



-- 
Born in Roswell... married an alien...
http://emptyhammock.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message