qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robbie Gemmell <robbie.gemm...@gmail.com>
Subject Re: Java Qpid Broker - flow to disk
Date Tue, 09 Apr 2013 16:44:25 GMT
Hi Michal,

There is no explicit flow-to-disk, but rather an implicit effect from the
way the messages are referenced. What the broker does is store persistent
messages on disk and reference them with soft references, such that if the
JVM needs to garbage collect them to make room for additional message data
it can. If the messages get consumed before that is necessary, the store is
effectivelly being used write-only. If the amount of data begins and the
structures necessary to reference the messages begins to exceed the heap
size then message references will need to be reclaimed, and at the point
the message is being used again (e.g sent to a consumer) the broker will
have to load it back from the store.

If using the BDB message store there is some additional caching on top of
this since BDB itself maintains a cache of the data which can use up to 60%
of the heap by default (configurable), such that even if the original
reference is collected the data can still be in memory at the time it is
request, aiding performance. If the amount of data begins to exceed the
cache size then BDB will begin to have to go to disk to satisfy some
requests, the exact behaviour of which becomes somewhat dependant on the
activity of the particular queues each message is on.

Effectively, the broker is limited by the number of persistent messages
(since the operational data structures are all kept in-memory even for
persistent messages) and the actual size of the messages is a secondary
issue and could greatly exceed the heap size. The issue with the size of
the data at this point would be the level of performance that can be
achieved whilst reading it back from disk as it is required, which is again
somewhat dependant on activity and the behaviourof the particular store in
question.

Robbie

On 8 April 2013 14:26, Michal Zerola <michal.zerola@gmail.com> wrote:

> Hello,
>
> I am curious what is the current state of the flow-to-disk implementation
> in
> Java Qpid broker. Especially, what happens if there are no consumers and
> cumulated size of persistent messages starts to reach the JVM memory limit.
>
> Thank you,
>
> Michal
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Java-Qpid-Broker-flow-to-disk-tp7591254.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message