qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kim van der Riet <kim.vdr...@redhat.com>
Subject Re: Help configuring persistence plugin
Date Mon, 31 Jan 2011 18:16:30 GMT
On Mon, 2011-01-31 at 12:01 -0300, Angel Freire wrote:
> Hi,
> I need a guide or help configuring the persistence plugin [0].
> This is my current qpidd.conf:
> log-to-syslog=yes
> log-to-stderr=no
> daemon=yes
> auth=no
> worker-threads=16
> # transport=tcp
> pid-dir=/var/run/qpidd
> data-dir=/var/lib/qpidd
> store-dir=/var/lib/qpidd/store
> num-jfiles32
> jfile-size-pgs=96
> wcache-page-size=128
> tpl-num-jfiles=32
> tpl-jfile-size-pgs=96
> tpl-wcache-page-size=128
> I need to be able to enqueue about 2.500.000 msgs that are normally
> 400 bytes long.
> Currently, with those settings, I get a lot of messages like this:
> framing-error: Enqueue capacity threshold exceeded on queue
> "charges_async_request". (JournalImpl.cpp:621)
> I'm using the version 0.6 of the broker and the plugin.
> Thanks,
> Angel
> [0] http://qpidcomponents.org/download.html#persistence

Your store size is too small. Using num-jfiles=32 (I hope you have an
'=' in there) and jfile-size-pgs=96 yields a total store size of 192 MiB
(201,326,592 bytes).

The unit for the jfile-size-pgs parameters is read cache pages of 64kiB
each, so 96 such units yields 96 * 64 * 1024 = 6291456 bytes (6.000
MiB). 32 such files brings the total to 192MiB.

Assuming your messages are 400 bytes each, and knowing that the store
saves messages in 128 byte blocks, each message will consume 4 blocks =
512 bytes of disk space. 2.5 million such messages will need 1.19GiB. I
recommend allocating twice this (but you can go as low as 1.5 times this
amount), so a store of about 2.4 GiB should be enough, provided you are
consuming more-or-less in the same order of production, and you are not
using transactions or frequent flushing.

Keeping 32 files (you can go as high as 64, but watch out for having
enough file handles available), this makes each file about 76.8MiB.
Since each unit of file size is 64kiB, this can be achieved using
jfile-size-pgs=1229. Note that with such large journal sizes, it will
take a noticeable amount of time to declare each queue - as 2.4GiB of
journal files need to be created and formatted - and also to recover in
the case of a broker restart.

The following resources may help from the persistence section in the
Qpid FAQ[1]:
How do I configure the persistence store?

[1] https://cwiki.apache.org/qpid/faq.html#FAQ-Persistence

Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

View raw message