activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Attila Nagy <...@fsn.hu>
Subject Long pauses with leveldb, no pauses with kaha
Date Wed, 04 Dec 2013 10:24:30 GMT
Hi,

I use ActiveMQ 5.9.0 on FreeBSD with java 1.7.0_25-b15 and native 
leveldb (default settings).
Four servers, fully meshed network of brokers, only two get effective load.

I publish from a small number of servers (around 30) to a virtual topic, 
then consume the aggregated messages from one STOMP client.
The average message size is around 250 bytes, the incoming message rate 
is around 8k+6k messages per second on the two servers.

What I see is the flow towards the consumer periodically pauses (maybe 
the incoming flow too, but I don't monitor that).
Measuring three of those pauses gave 34.01463485, 34.03087783 and 
53.02782011 seconds, during which no messages were delivered to the 
client at all. The first two are pretty suspicious.

Digging into it a little, "top" on the ActiveMQ server shows that one 
thread is active during the pauses, and it's mostly in pipe read state. WTF.
Doing a ktrace (strace-like FreeBSD tool) shows that it spawns multiple 
"ln" commands (yikes) which mess with the leveldb files (I guess it's 
the compaction job).

It seems others have performance issues with leveldb too (like 
http://symas.com/mdb/hyperdex/, maybe you should also switch to LMDB :) 
), so instead of playing with its tunable knobs (I've already failed 
with them, see my previous mail about kahadb being faster than leveldb), 
I've just switched to kahadb (also default settings) to check whether 
this causes the delays.

Yes, leveldb causes the long pauses in message deliveries. I'm running 
two hours since I've switched, and I haven't yet had a second with zero 
delivered messages, and the overall average rate of message deliveries 
per seconds is also smoothed out, no fluctuation.

Are you aware of this issue? Skimming through the list it seems there 
may be others hit by this, but I couldn't yet find an e-mail which would 
state that leveldb is bad for performance, you should a) avoid it, b) 
set these knobs to these, c) wait for the next revision, we'll fix this.

Thanks,

Mime
View raw message