activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hiram Chirino <hi...@hiramchirino.com>
Subject Re: Long pauses with leveldb, no pauses with kaha
Date Thu, 20 Feb 2014 02:06:30 GMT
Yep, at least I've noted that this can happen under consistent load.
I think part of the problem is that leveldb is periodically doing
level compactions which can cause pauses in it's write operations.

HyperDex and RocksDB are both similar libs which we could swap out
leveldb for in the future which seem to promise a more consistent
write rate.


On Wed, Dec 4, 2013 at 5:24 AM, Attila Nagy <bra@fsn.hu> wrote:
> Hi,
>
> I use ActiveMQ 5.9.0 on FreeBSD with java 1.7.0_25-b15 and native leveldb
> (default settings).
> Four servers, fully meshed network of brokers, only two get effective load.
>
> I publish from a small number of servers (around 30) to a virtual topic,
> then consume the aggregated messages from one STOMP client.
> The average message size is around 250 bytes, the incoming message rate is
> around 8k+6k messages per second on the two servers.
>
> What I see is the flow towards the consumer periodically pauses (maybe the
> incoming flow too, but I don't monitor that).
> Measuring three of those pauses gave 34.01463485, 34.03087783 and
> 53.02782011 seconds, during which no messages were delivered to the client
> at all. The first two are pretty suspicious.
>
> Digging into it a little, "top" on the ActiveMQ server shows that one thread
> is active during the pauses, and it's mostly in pipe read state. WTF.
> Doing a ktrace (strace-like FreeBSD tool) shows that it spawns multiple "ln"
> commands (yikes) which mess with the leveldb files (I guess it's the
> compaction job).
>
> It seems others have performance issues with leveldb too (like
> http://symas.com/mdb/hyperdex/, maybe you should also switch to LMDB :) ),
> so instead of playing with its tunable knobs (I've already failed with them,
> see my previous mail about kahadb being faster than leveldb), I've just
> switched to kahadb (also default settings) to check whether this causes the
> delays.
>
> Yes, leveldb causes the long pauses in message deliveries. I'm running two
> hours since I've switched, and I haven't yet had a second with zero
> delivered messages, and the overall average rate of message deliveries per
> seconds is also smoothed out, no fluctuation.
>
> Are you aware of this issue? Skimming through the list it seems there may be
> others hit by this, but I couldn't yet find an e-mail which would state that
> leveldb is bad for performance, you should a) avoid it, b) set these knobs
> to these, c) wait for the next revision, we'll fix this.
>
> Thanks,



-- 
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino

Mime
View raw message