qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kim van der Riet <kim.vdr...@redhat.com>
Subject Re: how to resolve Enqueue capacity threshold exceeded error
Date Wed, 15 Apr 2009 12:15:11 GMT
On Wed, 2009-04-15 at 09:44 +0100, Gordon Sim wrote:

> I.e. you need at least 272,000,000 bytes capacity. The default 'page 
> size' is 32k so each file will be 128*32k=4096k, and 64 files of that 
> size would give you 262,144,000. I think you may run out of space even 
> with those settings.

The file size parameter (as specified in the help) is in read pages,
each of which is 64k (the write page cache is 32k). A setting of 128
will yield a file of 8 MiB each. Using 64 files (the current max because
of file handle limit concerns) will give a total per-queue file capacity
of 512 MiB. This is going to take time to initialize  - probably several
seconds each time a queue of this size is declared.

The current file count limits are 4-64, the current file size limits are
1-32768 (which translates to 64 kiB - 2 GiB each). Combined, the current
store size limit is 128 GiB per queue.

Here is a brief explanation of the workings and reasoning behind this

The store creates a circular buffer from the files. Because the process
of dequeueing records involves writing a dequeue record which logically
invalidates a previous enqueue record (rather than literally erasing an
old record), a full store is a fatal condition because neither enqueues
nor dequeues may take place. For this reason, an approximately 80% full
limit is set on enqueues - the idea being that when the store is this
full, allowing only dequeues can free up space which will eventually
allow enqueues to continue.

A further rule of the circular buffer is that no file that contains any
still enqueued record may be written, even if the enqueued record is
right at the end of the file. This safeguards the integrity of the data
in the store - as crucial recover information would be overwritten in
the file header as soon as the next write in that file occurs.

The mechanism employed by the store to check this limit is to look ahead
from the current write position plus the proposed write size in the
store by 20% of the total capacity of the journal and make sure that the
file at this position is writable - ie contains no enqueued records. As
soon as this test fails, the Enqueue Capacity exception is thrown to the
enqueueing application. Because the entire file at the +20% position
must be free of enqueues, the exact mathematical percentage of the
journal's capacity this represents can vary depending on the size of the
files, their number and where the last record is located in the file.
For a large number of files (such as the 64 you are now using), this
will be fairly close to 80%, but for a small file count, this can be as
low as 50%.

Correctly sizing the store up-front is obviously important. To do so, it
is necessary to correctly estimate the maximum queue depth in bytes. As
a rough rule of thumb, the journal should be at least twice the maximum
anticipated queue depth. (If disk space and/or queue declare latencies
are critical, it is possible to reduce this if the parameters are
carefully controlled.) Being a circular buffer, however, it is also
important that the message consumers be well-behaved - ie consumers
acknowledge the messages they have received promptly. If a consumer
requests a message, but delays in acknowledging it after receiving it,
then the store must hold on to that message, and it can remain long
after the messages around it have been dequeued. This will eventually
cause the store to throw the enqueue threshold exception because this
remaining record cannot be overwritten until its receipt has been
acknowledged by the consumer. Gordon - you may be able to clarify this
further in API terms better.

Rather a long explanation, but I hope it will shed some light on the
store and its limitations.


Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

View raw message