activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From davidmc <aranman...@gmail.com>
Subject Re: AMQCPP Openwire much slower than Stomp?
Date Thu, 19 Apr 2007 02:45:55 GMT

Hello,

Your problem really looks caused by a TCP_NODELAY issue.
I had to do research to solve a similar problem in ActiveMQ, so I thought I
could try and save you some time.

The root of this kind of problem is the interaction between Nagle's
algorithm and Delayed ACK timeouts. For a good explanation, check these
links:
http://en.wikipedia.org/wiki/Nagle_algorithm
http://developers.slashdot.org/comments.pl?sid=174457&threshold=1&commentsort=0&mode=thread&cid=14515105
http://www.port80software.com/200ok/archive/2005/01/31/317.aspx
Summarizing:
-Nagle's algorithm keeps your packet in the socket buffer until the buffer
is filled or you get an ACK back.
-Delayed ACK algorithm on the other side of the socket, doesn't send an ACK
back because it thinks it should wait for some real information to accompany
the ACK with (to improve throughput). However the other side doesn't want to
send anything; so everything blocks until the Delayed ACK timeout kicks in,
and the other side sends an ACK anyway because it's not going to wait
forever.

My case was exactly as yours:
-the behaviour is different between a local broker and a remote broker,
because:
*Windows sockets that connect to localhost don't seem to be really sockets,
or either, the TCP behaves differently (no Nagle's algorithm).
*Linux localhost sockets also behave a bit differently from Linux remote
sockets when it comes to Nagle / Delayed ACK issues.
-the behaviour is different in Linux and in Windows, because
*in windows the Delayed ACK timeout is 200ms. So if your broker is in
Windows, you will notice messages with a roundtrip (latency) time of 200ms.
*in Linux this timeout is 40ms "only".
-message size matters, because big messages fill the buffer and get sent
immediately regardless of ACKs received or not.
Also I have noticed some strange behaviour with very little packets (less
than 512 or so bytes), with those packets, disabling Nagle might not work.
The size below which problems happen is different in a Linux local scenario
than in a Linux remote scenario.

How to check if this is really the problem:
-Go to AMQCPP source code and add a line to your socket creation code, where
you explicitely set TCP_NODELAY to True always (to disable Nagle's
algorithm). Rebuild and compare.
-Capture traffic with Wireshark (former Ethereal), or tcpdump, and see if
the broker takes a fixed time (40ms in Linux or 200ms in Windows) to send
ACKs back.

Hope this helps :)

David


Albert Strasheim wrote:
> 
> Hello all
> 
> On Wed, 18 Apr 2007, Nathan Mittler wrote:
> 
>> Unfortunately we didn't do a performance analysis, as we've been focused
>> on
>> just getting 2.0 out the door.  We did however notice a bit of slowness
>> in
>> our integration tests.  Have you, by any chance, profiled the openwire
>> connector to see where it's spending its time?
>> 
>> With any luck, we can zero in on the bottlenecks quickly and cut a 2.1.
> 
> I'm going to spend some time on this over the next few days. The 
> symptoms are quite strange. Factors that seem to influence the speed 
> include:
> 
> - whether connection is to a local broker or a remote broker
> - whether the broker is running on Windows or Linux
> - message size possibly in combination with wire format
> 
> I'll report back if I figure out anything more useful.
> 
> Regards,
> 
> Albert
> 
> 

-- 
View this message in context: http://www.nabble.com/AMQCPP-Openwire-much-slower-than-Stomp--tf3599986s2354.html#a10070798
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.


Mime
View raw message