activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Kluesing>
Subject OOM with high KahaDB index time
Date Mon, 18 Jan 2010 22:14:35 GMT

I'm running the 5.3 release as a standalone broker. In one case, a producer is running without
a consumer, producing small, persistent messages, with the FileCursor pendingQueuePolicy (per option and flow control memoryLimit set
to 100mb for the queue in question. (Through a policy entry) 

As the queue grows above 300k messages, KahaDB indexing starts climbing above 1 second. At
around 350k messages, the indexing is taking over 8 seconds. At this point, I start getting
java out of heap space errors in essentially random parts of the code. After a while, the
producers timeout with a channel inactive for too long error, and the entire broker basically
wedges itself. At this point, consumers are generally unable to bind to the broker quitting
with timeout errors. When they can connect, consuming a single message triggers an index re-build,
which takes 2-8seconds. Turning on verbose garbage collection, the jvm is collecting like
mad but reclaiming no space. 

If I restart the broker, it comes back up, I can consume the old messages, and can handle
another 350k messages until it wedges.

I can reproduce under both default gc and incremental gc. 

Two questions:
- It seems like someone is holding onto a handle to the messages after they have been persisted
to disk - is this a known issue? Should I open a JIRA for it? (Or is there another explanation?)

- Is there any documentation about the internals of KahaDB - the kind of indices etc? I'd
like to get a better understanding of the index performance and in general how KahaDB compares
to something like BerkeleyDB.


View raw message