qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "xiaodan.wang" <xiaodan.w...@salesforce.com>
Subject Re: Possible for unconsumed messages to prevent GC from reclaiming memory held by prior messages?
Date Wed, 29 Oct 2014 22:32:23 GMT
Hi Rob, apologies for taking so long to follow-up. We now have meaningful
data (both a reproducible test and heap dump) to describe the issue. I will
first describe the setup of the test we used to reproduce this issue and
then Helen, on of my teammate, will describe the heap dump analysis.

To recap, we observed that the Qpid (Java) broker heap gradually trended up
in production when we had a handful of messages on a special "suspend" queue
that were unconsumed. Even though we found no other messages when we
inspected the broker, attempts to GC and reclaim broke memory failed. That
is until we manually deleted the handful of suspended/unconsumed messages.

I will now describe the test we used to reproduce this issue. We are using
non-persistent messages, asynchronous onMessage delivery, and transacted
sessions. We also ran the test using both the v0.16 Java broker and the
v0.32/trunk Java broker and verified that the issue occurs on both.

1) We setup two queues, A and B.

2) We first enqueue one message to queue B and do not consume the message
(this message remains unconsumed for the entire duration of the test)

3) Next we setup a single test thread that runs iteratively and does:
   a) First enqueues 1000 messages to queue A and commits
   b) Next, checks out a NEW qpid session and creates a consumer for queue
A. Invokes consumer.receive to fetch one message and then immediately rolls
back the receive.
   c) Close the consumer for queue A
   d) Using the SAME qpid session from step b), create a consumer for queue
B. Invoke consumer.receive and to fetch one message and then immediately
rolls back the receive.
   e) Close the consumer for queue B and close the qpid session.
   f) Create a NEW qpid session, and create a consumer for queue A.
   g) Invoke consumer.receive repeatedly to drain all 1000 messages from
queue A and then commit.
   h) Close the qpid session, at this point there are no messages on queue A
and a single message on queue B.

4) Repeat step 3) many times

5) Before test is finished, validate that queue B has a single message and
queue A is empty.

At the conclusion of the test, we observed that Qpid broker heap is nearly
exhausted and memory cannot be reclaimed via GC. Also, we can reproduce the
same issue if steps 3b) - 3e) is replaced with session.createBrowser (queue
browse/peeking) instead of consumer.receive on queues A and B.

Helen is in the process of writing up the heap dump analysis and will post
Thanks! Dan

View this message in context: http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615932.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

View raw message