httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Pratley <n...@dotsec.com>
Subject [users@httpd] mod_proxy intermittent "(70007)The timeout specified has expired"
Date Thu, 03 Sep 2009 04:14:24 GMT
Hi,

I have two web servers that proxy around 150 backend sites, and are
experiencing a lot of mod_proxy timeout errors. The two servers are Xen
guests running CentOS 5.3, and sit behind an LVS-NAT load balancer which
is also a Xen domU, also running CentOS 5.3. The errors in the error log
are:

[error] (70007)The timeout specified has expired: proxy: HTTP: attempt
to connect to x.x.x.x:80 (www.somesite.com) failed
[error] ap_proxy_connect_backend disabling worker for (www.somesite.com)
[error] proxy: HTTP: disabled connection for (www.somesite.com)

The last error above appears every time a request is made for
www.somesite.com until that backend site is re-enabled by mod_proxy. The
errors affect a number of different backend sites with the only common
factor appearing to be that these are among the most popular
reverse-proxied sites. The errors don't seem to appear for the less
popular backend sites.

Something I've observed a number of times is that a TCP connection to a
particular backend site will be stuck in the SYN_SENT state for a long
time (approx. 2 minutes), and as soon as that connection disappears the
timeout error occurs and the backend site is disabled.

Until last week this service was running on a single domU with no load
balancer, on completely different hardware. In this configuration the
timeout errors had occurred however they were nowhere near as consistent
- about 2 or 3 times a month compared to at least once per hour that
we're experiencing now.

We receive an average of around 300,000 hits per hour, up to about a
million during busy parts of the day (as reported by AWStats). We have a
10Mbps internet link however our cacti graphs show the average usage as
being around 2Mbps, peaking to 3Mbps, so it's unlikely that we're
saturating the network link.

Here is an example virtual host directive from our configuration files:

--------
<VirtualHost *:443>
    ServerAdmin admin@ourcompany.com
    DocumentRoot /var/www/html
    ServerName "www-somesite-com.ourcompany.com"
    LogLevel warn
    ServerSignature Off

    SSLEngine on
    SSLProtocol all -SSLv2
    SSLCipherSuite HIGH:+MEDIUM:!EXP:!SSLv2:!LOW
    SSLCertificateKeyFile /etc/httpd/certs/sp.key
    SSLCertificateFile /etc/httpd/certs/sp.cer
    SSLCertificateChainFile /etc/httpd/certs/server-chain.crt

    ProxyPass / http://www.somesite.com/
    ProxyPassReverse / http://www.somesite.com/
    ProxyHTMLURLMap http://www.somesite.com /

    SetOutputFilter line-editor
    RequestHeader unset Accept-Encoding
    ProxyPassReverseCookieDomain .somesite.com .ourcompany.com

    SetEnv        LineEdit
"application/x-javascript;text/plain;text/css;text/javascript;text/html"
    LERewriteRule "\"http\://www\.somesite\.com/" "\"/" R
    LERewriteRule "\'http\://www\.somesite\.com/" "'/" R

    Include conf.d/shib.conf.include

    ErrorDocument 502
https://sp.ourcompany.com/errors/providerSiteError_502.html

    SetEnv proxy-nokeepalive 1

</VirtualHost>
--------

Here are some more details of the web server:

# uptime
 14:04:05 up  5:28,  1 user,  load average: 0.00, 0.01, 0.02

# uname -a
Linux ckn-sp-2 2.6.18-128.4.1.el5xen #1 SMP Tue Aug 4 20:51:12 EDT 2009
x86_64 x86_64 x86_64 GNU/Linux

# free
             total       used       free     shared    buffers     cached
Mem:       4194304     946272    3248032          0      32312     127636
-/+ buffers/cache:     786324    3407980
Swap:      2031608          0    2031608

# /usr/sbin/httpd -v
Server version: Apache/2.2.3
Server built:   Jul 14 2009 06:02:39

# netstat -s
Ip:
    2934756 total packets received
    0 forwarded
    0 incoming packets discarded
    2934742 incoming packets delivered
    3207218 requests sent out
Icmp:
    2267 ICMP messages received
    10 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 286
        redirects: 1516
        echo requests: 465
    773 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 308
        echo replies: 465
IcmpMsg:
        InType3: 286
        InType5: 1516
        InType8: 465
        OutType0: 465
        OutType3: 308
Tcp:
    61435 active connections openings
    110304 passive connection openings
    36 failed connection attempts
    2578 connection resets received
    86 connections established
    2777064 segments received
    3011011 segments send out
    40041 segments retransmited
    0 bad segments received.
    318 resets sent
Udp:
    157061 packets received
    264 packets to unknown port received.
    0 packet receive errors
    157337 packets sent
TcpExt:
    3 invalid SYN cookies received
    36 resets received for embryonic SYN_RECV sockets
    39 packets pruned from receive queue because of socket buffer overrun
    2 ICMP packets dropped because socket was locked
    95627 TCP sockets finished time wait in fast timer
    735 time wait sockets recycled by time stamp
    53736 delayed acks sent
    28 delayed acks further delayed because of locked socket
    Quick ack mode was activated 1415 times
    37023 packets directly queued to recvmsg prequeue.
    1568 packets directly received from backlog
    414402 packets directly received from prequeue
    835894 packets header predicted
    25303 packets header predicted and directly queued to user
    641977 acknowledgments not containing data received
    491841 predicted acknowledgments
    653 times recovered from packet loss due to fast retransmit
    129 times recovered from packet loss due to SACK data
    11 congestion windows fully recovered
    TCPDSACKUndo: 39
    35 congestion windows recovered after partial ack
    44 TCP data loss events
    104 timeouts after reno fast retransmit
    29 timeouts after SACK recovery
    8 timeouts in loss state
    1044 fast retransmits
    22 forward retransmits
    779 retransmits in slow start
    5787 other TCP timeouts
    TCPRenoRecoveryFail: 19
    3 sack retransmits failed
    1 times receiver scheduled too late for direct processing
    1744 packets collapsed in receive queue due to low socket buffer
    1227 DSACKs sent for old packets
    144 DSACKs received
    124 connections reset due to unexpected data
    689 connections reset due to early user close
    3991 connections aborted due to timeout
    10 times unabled to send RST due to no memory
IpExt:


----

All ideas are welcome.

Thanks,
Nick

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message