qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakub Scholz <ja...@scholz.cz>
Subject Paged queues questions (C++ broker)
Date Wed, 06 Nov 2013 23:09:21 GMT
Hi Gordon & everyone else,

I finally managed to spent some quality time with the queue paging today. I
have quite a lot of questions / comments / feedback. I hope you don't mind.
If you want, I can as usually rise the JIRAs or provide more details for
reproducing of the problems. Just let me know what is a real problem and a
JIRA should be raised and what "works as designed" and no JIRA is needed.

Thanks & Regards
Jakub

---

1) Why is the page file created as executable?
-rwx------. 1 qpidd qpidd 110592000 Nov  6 19:07 my-test-queue-2

---

2) As I understood from the previous discussion, the queue can store only
messages which are not bigger than the page size (i.e. page_factor * memory
page). Does the value of the page_factor have also some other implications?
Does it have some effect on performance?

---

3) Can I somewhere see some statistics about how many pages are currently
in memory or how many are off-loaded to the disk? The qpid-tool doesn't
seem to have this information.

---

4) It would be really great to have a possibility to change the
qpid.max_pages_loaded parameter without deleting and recreating the queue.
Even having this as an "offline" option - i.e. when the change would be
possible only when the broker is switched ogg - would be IMHO useful.

---

5) It would be great to have a broker configuration option "--paging-dir"
to be able to specify the directory for the page files should be created.
Right now it seems to use the "--data-dir" parameter.

---

6) Even when using the paging, the broker memory consumption seems to
increase all the time. Is it just some sort of "index" linking to the
offloaded pages? If it is only the index what is kept in memory, does that
mean that with higher qpid.page_factor I will have lower memory consumption?

---

7) The purge QMF method doesn't seem to work. When I call it for the first
time, it seem to always purge only the messages from the loaded pages. When
I try to call it again I get the following error:
Connection closed by peer with amqp:internal-error: Queue
"my-test-queue-2": Dequeuing message with null persistence Id.
(/home/jakub/qpid/qpid-trunk/cpp/src/qpid/legacystore/MessageStoreImpl.cpp:1372)

(the queue configuration: durable=true, max_pages_loaded=1, page_factor=1).
In fact with the queue configured as above, this happens to receiver as
well - when reading the queue for the first time it gets only the messages
from the loaded page. When read for the second time, it creates the same
error.

I also tried to change the values of max_pages_loaded and page_factor ...
but that doesn't seem to change much. I can read/purge more messages before
seeing the error. But it still seems to occur regularly.

---

8) When the queue is created as above (max_pages_loaded=1, page_factor=1),
but as non-durable, purging and reading still doesn't work properly. It
always reads / purges only the loaded pages. But of course I do not get the
error from the persistent message store. However, with increase
max_pages_loaded and/or page_factor (e.g. 10 and 10), the problem seems to
be resolved (for non-persistent queues).

It seems to me that when the pages are too small / only few pages are
loaded, the broker doesn't manage to load the pages quickly enough from
disk and that some kind of "starves" the receiver. Perhaps setting minimal
values for the page_factor / max_pages_loaded can help here?

---

9) I know that the paged queues don't support LVQ functionality and
probably also message priorities. But it seems that paged queues don't
really support filters / selectors as well, or? The client using selector
always receives only the messages which match the selector and are in the
loaded pages. I guess it makes sense - since the broker doesn't keep
complete message headers, it cannot easily match
the selectors. But in that case, I would suggest to raise an error if a
consumer with filter/selector is created.

---

10) When I try to send a message bigger than the page size, I get following
error:
Link detached by peer with amqp:precondition-failed: precondition-failed:
Message is larger than page size for queue backed by
/home/qpidd/qpid/0000/data/broker/my-test-queue-2
(/home/jakub/qpid/qpid-trunk/cpp/src/qpid/broker/PagedQueue.cpp:107)

Is it intentional to include the full path to the page file? I would
definitely prefer to show just the file name or a queue name.

---

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message