activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From davidmc <aranman...@gmail.com>
Subject High latency for small messages problem
Date Fri, 02 Feb 2007 09:33:35 GMT

Hello,

We are evaluating the possible use of ActiveMQ in our projects and have
encountered this problem.

When sending small messages (for example, ByteMessages with a 1KB byte
array) from a producer to a consumer through a broker, there is much higher
latency than with bigger messages (for example, ByteMessages with a 32KB
byte array), using the TCP transport.
Changing the socketBufferSize parameter, the tcpNoDelayEnabled flag does not
help.
Version of ActiveMQ is 4.1, and in 4.0 there is also this problem.

Our test is as follows: one JVM is the broker, the other JVM runs a producer
and a consumer in separate threads (with separate connections and sessions).
We are using a queue, no persistence, no transactions,
Session.AUTO_ACKNOWLEDGE.
The producer sends a message just after taking note of the time with
System.nanoTime(), and then waits on a semaphore.
When the consumer receives the message (implementing the onMessage() method
of the MessageListener interface), the first thing that is done is taking
notice of the time with System.nanoTime().  The difference in time is
computed and written to a file, after which the consumer releases the
semaphore so that the producer sends the next message.

With 1 machine running Red Hat EL4 Update 3, hosting both the broker process
and the producer/consumer process, we get a latency of 39ms with 1KB
ByteMessages and latency of 0-1ms with 32KB ByteMessages. The frontier for
latency appearing seems to be around 16KB.
With 1 machine running WinXP Pro SP2, hosting both the broker process and
the producer/consumer process, we get no latency (0-1 ms) with messages of
any size between 1KB and 32KB.
With 2 machines in a LAN, no matter the combination of WinXP / Linux (even 2
WinXP machines) we get high latency (even 200 ms for 2 WinXP machines) with
1KB messages and much lower latency (not 0, but low) with 32 KB messages.

Some more facts:
The first 1KB message has high latency (as expected due to initialization of
stuff), the next 8-20 have no latency, and then the rest have high latency.
In Linux localhost case, if the producer waits 1 second between sending 1KB
messages, latency is low. Have not tried in other scenarios.
Normally we do tests restarting the broker each time. But, if we do not
restart the broker, the results can be more strange. Sometimes, there is a
pattern where a 1KB message has no latency, the next had 39ms latency, the
next has no latency, the next has 39ms latency, etc. In one of these times
we took an Ethereal trace and found that the producer was sending 2 messages
in the same TCP packet of around 2KB size.

The broker configuration file is:
http://www.nabble.com/file/6205/activemq.xml activemq.xml 

The code to perform the test is:
http://www.nabble.com/file/6207/LatencyTest.java LatencyTest.java 

Here are some logs we have taken:
http://www.nabble.com/file/6206/logs.zip logs.zip 
For 1KB messages you also have ethereal logs (32KB message ethereal logs are
too big files).
In every case, the file "client.txt" is the log of the producer/consumer
process, the file "broker.txt" is the log of the broker process. Both these
logs use the log4j logger of ActiveMQ.
The "nolog" folders have logs without use of the log4j logger, since
printing lots of messages disturbs a bit the results.
Then you have a file called for example 1BytesMessage.txt (stands for 1KB
messages), with the latency times and time of receiving by the consumer.

All this is very strange... we do not know if the problem lies in the Linux
TCP stack, the Java I/O or networking libraries, ActiveMQ code, some lack of
proper configuration... we believe however that it is the producer which is
holding sometimes a message before sending the packet to the broker.
-- 
View this message in context: http://www.nabble.com/High-latency-for-small-messages-problem-tf3159901.html#a8764113
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message