httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Feldhacker, Chris" <Feldhacker.Ch...@principal.com>
Subject RE: [users@httpd] Slow performance of Apache on Windows
Date Thu, 10 Mar 2011 17:58:57 GMT
-----Original Message-----
From: William A. Rowe Jr. [mailto:wrowe@rowe-clan.net] 
Sent: Wednesday, March 09, 2011 4:46 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Slow performance of Apache on Windows

> On 3/9/2011 4:44 PM, William A. Rowe Jr. wrote:
>> On 3/9/2011 3:15 PM, Feldhacker, Chris wrote:
>>>
>>> It was still odd that the problem only seemed to occur from our internal server,
HTTP downloads from the internet seemed fine.  Out of curiosity we setup a Windows Server
2008 running Apache, and a RedHat Linux server running Apache.  The performance of the 2008
server was the same as the 2003 server, but the performance of the Linux server was double
that of the Windows machines (10-11MB/s)!  In fact, not only did the Windows 7 client maintain
10-11MB/s, but the Windows XP client had the same 10-11MB/s performance as well!
>> 
>> On this one issue, did you try 'Enable Sendfile off' on the windows box?
> Sorry, asked and answered in the original summary, I just missed it the first time through.


We did get this problem resolved -- you were right.

If you'd like the tail-between-our-legs explanation and hopefully to benefit anyone else that
might encounter this problem:
Ultimately, we did not discover the solution sooner due to inconsistent testing and shortcuts.
 (Isn't that how it usually goes?)  Most testing was done in an isolated lab (especially network
captures), but sometimes we'd play with settings on the production network out of convenience.
 And since the problem occurred with both XP and W7, we would try different things with different
clients.  And, once we made a change and it didn't work, we would set things back to "default"
and go on.

We did try setting the three Apache options for windows:  EnableSendfile Off, EnableMMAP Off,
Win32DisableAcceptEx.  But we must have done this on the production network and tested with
the XP client, which only got the expected slow(er) result.  However, apparently the EnableSendfile
directive really did make a difference, we just didn't realize it (we can only assume the
production network must have been under some load at the time).  So, we removed the settings
back to default and kept searching...

Per your suggestion, we focused on this setting again in the isolated lab -- the XP client
achieved the 10MB/s, as did the W7 client.  Nice.

Before finding this solution we were giving additional focus to the network traces (after
all, the answer must be there, right?).  In case anyone has interest, focusing first on traces
from the Windows 7 client:
The TCP window size are slightly different, the client was always consistent at 65700, but
Apache=64099, IIS=64094, SHTTPS=64094.  (Don't know why Apache is 5 bytes higher, but anyway...)
 The window size really doesn't seem to matter.  Looking at the traffic patterns, the server
sends 2 packets, and then the client ACKs the last packet sent.  But sometimes the server
must be sending packets fast enough that 3 packets get through and the client ACKs the last
packet sent.  So, sometimes the client is ACKing every other packet, sometimes it's every
3rd packet.  Which means sometimes the last packet from the server with the Push flag can
be 1st, 2nd, or 3rd in the final "set".  When the packet is 2nd or 3rd, the client has no
problems and an ACK is sent right away with very little delay.  When the packet is 1st, then
(with Apache) the server waits for an ACK from the client before continuing.  Of course, this
is where Nagel's algorithm comes into play, which explains the client delay (looking at the
traces, it varies between 35-210ms).

I say I don't think the window size matters because the same traffic pattern appears with
IIS and SHTTPS -- sometimes the W7 client ACKs every 3rd packet instead of every other packet,
so it can still end up on an "odd" boundary.  The difference being, with IIS and SHTTPS, when
the last packet with the Push flag comes through and if it happens to be 1st, *the server
starts on the next sequence and keeps sending packets without stopping to wait for the client
ACK*.  So, IIS and SHTTPS are continuously streaming without waiting for ACKs of the Push
packets, but Apache stops and waits.
(You would think using the TCP_NODELAY option on the client should workaround this problem.
 We definitely tested this previously with no luck, so we double-checked to make sure and
still no luck.  We tested this using curl with the "--tcp-nodelay" option, so unless there
are issues with curl, I'm not sure why this doesn't seem to result in any improvement...)

So, what does the traffic look like with a W7 client against Apache with the EnableSendfile
Off directive?  Client ACKs are still sent after every 2 to 3 packets, but when the packet
with the Push flag is sent Apache no longer stops and waits for an ACK but continues to stream
packets -- just like IIS and SHTTPS did.  No only that, but Push packets appear a LOT more
frequently, which must mean Apache is streaming content in smaller chunks.  There are 5 regular
packets and the 6th contains the Push (8192 bytes total).  With EnableSendfile On, there are
44 regular packets and the 45th contains the Push (65536 bytes total).
I don't know what Apache is doing differently with the EnableSendfile Off, but it seems to
resolve the performance issue by continuously streaming data despite apparently using a smaller
buffer size.
(Could probably squeeze an additional 6% performance improvement if the buffer size was larger
so that every 6th packet wasn't only 60% full, but at this point I'm not complaining!)

It's hard to draw any conclusions from the XP traces -- time precision of the XP machine must
be pretty poor as almost all of the time deltas display as zero; there's occasionally a 15ms
blip with no apparent pattern, I assume it just happens to be whenever the clock happens to
tick.  However, the XP client always ACKs every 1-2 packets, even mid-stream; there is never
a sequence of 3 packets.  It's almost like the XP client is configured to ACK every packet
rather than every other one (strange, nothing in the registry suggests this...perhaps something
under a different key).  So, with EnableSendfile On, it looks like Apache is still waiting
for an ACK, but because it appears the client is trying to ACK every packet there is no delay.
 With EnableSendfile Off, the traffic appears the same, but again it's really hard to draw
any conclusions given the low precision...


So, there you have it.  
I think it's really unfortunate that a default install of Apache on Windows has this issue
-- and because the symptoms vary so drastically based on the client OS, it's very easy to
get mislead and think the problem is with the client...or the application...or the server...or
the network.

Does anyone know if there is even a valid scenario when EnableSendfile MUST be turned On on
a Windows machine?  Seems like Off should be the default on Windows...

Thanks for the feedback and helping us double-check our work!


-----Message Disclaimer-----

This e-mail message is intended only for the use of the individual or
entity to which it is addressed, and may contain information that is
privileged, confidential and exempt from disclosure under applicable law.
If you are not the intended recipient, any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify us immediately by
reply email to Connect@principal.com and delete or destroy all copies of
the original message and attachments thereto. Email sent to or from the
Principal Financial Group or any of its member companies may be retained
as required by law or regulation.

Nothing in this message is intended to constitute an Electronic signature
for purposes of the Uniform Electronic Transactions Act (UETA) or the
Electronic Signatures in Global and National Commerce Act ("E-Sign")
unless a specific statement to the contrary is included in this message.

While this communication may be used to promote or market a transaction
or an idea that is discussed in the publication, it is intended to provide
general information about the subject matter covered and is provided with
the understanding that The Principal is not rendering legal, accounting,
or tax advice. It is not a marketed opinion and may not be used to avoid
penalties under the Internal Revenue Code. You should consult with
appropriate counsel or other advisors on all matters pertaining to legal,
tax, or accounting obligations and requirements.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message