qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Helen Kwong <helenkw...@gmail.com>
Subject Re: Unconsumed messages can still prevent GC of consumed messages
Date Wed, 04 Nov 2015 23:53:25 GMT
Hi Rob, will the fix be committed to the 0.32 branch as well?

On Wed, Nov 4, 2015 at 3:11 PM, Rob Godfrey <rob.j.godfrey@gmail.com> wrote:

> Hi Helen,
>
> glad to hear the patch worked for you, we'll make sure that fix is in the
> upcoming 6.0 release (hopefully the Beta for that will be out by the end of
> this week),
>
> Cheers,
> Rob
>
> On 4 November 2015 at 21:57, Helen Kwong <helenkwong@gmail.com> wrote:
>
> > Hi Rob,
> >
> > The patch fixed the issue in our tests. It looks like we ran into the
> > problem because we regularly create and close QueueBrowsers sharing the
> > same session to our set of queues, and while we constantly consume
> messages
> > from most queues with MessageConsumers, there can be a small number of
> > queues that we do not create any regular consumer to for a long time.
> >
> > Thanks again for fixing this so quickly (as usual)!
> >
> > Helen
> >
> >
> > On Sun, Nov 1, 2015 at 4:44 PM, Helen Kwong <helenkwong@gmail.com>
> wrote:
> >
> > > Thanks a lot for looking into this and fixing it so quickly! Will try
> the
> > > patch and let you know.
> > >
> > > Helen
> > >
> > > On Sat, Oct 31, 2015 at 4:50 PM, Rob Godfrey <rob.j.godfrey@gmail.com>
> > > wrote:
> > >
> > >> I've raised a jira https://issues.apache.org/jira/browse/QPID-6820
> > >> covering
> > >> the issue with state change listeners not being removed - I've also
> > >> attached a patch which I believe should fix the issue.
> > >>
> > >> The issue will only occur where a consumer runs out of credit while
> > there
> > >> are still messages on the queue, and then that consumer is closed...
> and
> > >> the first (oldest) message it was unable to consume is not itself
> > consumed
> > >> (or even delivered) to any other consumer.  The only way to free the
> > >> association is for the message to be acquired by some other party (it
> > >> doesn't need to be dequeued, just acquired - a session that started,
> > >> opened
> > >> a consumer, prefetched all the messages from the queue, and then
> closed
> > >> would be enough to unjam it).
> > >>
> > >> Basically the state listener is there in case the reason the consumer
> > was
> > >> unable to consume it was because it was a particularly large
> message...
> > if
> > >> it is subsequently acquired by a different consumer, the first
> consumer
> > >> needs to be woken up to look at subsequent messages on the queue to
> see
> > if
> > >> it can accept them.
> > >>
> > >> Let me know if you have a chance to test the patch, and if it solves
> > this
> > >> issue for you.
> > >>
> > >> Again, apologies that you've run into another obscure issue - but
> thanks
> > >> for taking the time to work out what was going on - much appreciated!
> > >>
> > >> -- Rob
> > >>
> > >> On 31 October 2015 at 07:55, Rob Godfrey <rob.j.godfrey@gmail.com>
> > wrote:
> > >>
> > >> > Hi Helen,
> > >> >
> > >> > thanks for the detailed analysis  - I'll look into this today.
> > >> >
> > >> > -- Rob
> > >> >
> > >> > On 31 October 2015 at 01:26, helenkwong <helenkwong@gmail.com>
> wrote:
> > >> >
> > >> >> Hi Rob,
> > >> >>
> > >> >> We've been using the v0.32 Java broker and haven't run into memory
> > leak
> > >> >> problems for a while, but we've just seen something similar to
what
> > we
> > >> saw
> > >> >> before happen again and would appreciate your help. See our
> previous
> > >> >> thread
> > >> >> here (from almost exactly a year ago):
> > >> >>
> > >> >>
> > >>
> >
> http://qpid.2158936.n2.nabble.com/Possible-for-unconsumed-messages-to-prevent-GC-from-reclaiming-memory-held-by-prior-messages-tp7615368p7615933.html
> > >> >> .
> > >> >>
> > >> >> The pattern, like before, is that when there's an unconsumed
> message
> > >> that
> > >> >> has been sitting in a Queue A for a long time, we can end up being
> > >> unable
> > >> >> to
> > >> >> reclaim old messages in another Queue B, even though these messages
> > >> have
> > >> >> already been consumed long ago. The link seems to be that at some
> > >> point in
> > >> >> the past, a consumer of Queue A and a consumer of Queue B shared
> the
> > >> same
> > >> >> ServerSession. And although the session and both consumers were
> > closed
> > >> >> long
> > >> >> ago, Queue A's head entry still refers to the old Queue A consumer
> > via
> > >> its
> > >> >> _stateChangeListeners list. Consequently, we're still holding
onto
> > the
> > >> old
> > >> >> messages that were in Queue B, via the common session.
> > >> >>
> > >> >> More specifically, the screenshot below shows how we go from a
> Queue
> > A
> > >> >> consumer referring to old Queue B messages: QueueConsumerImpl
->
> > >> _target
> > >> >> ->
> > >> >> _session -> commands table -> MessageTransfer belonging
to a Queue
> B
> > >> >> message
> > >> >> consumed long ago -> completionListener -> old PriorityQueueEntry
> of
> > >> >> Queue B
> > >> >> (highlighted in screenshot) -> many more old queue entries.
> > >> >> <
> > >> >>
> > >>
> >
> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_queueAConsumerToQueueBEntry.png
> > >> >> >
> > >> >>
> > >> >> But the Queue A QueueConsumerImpl was created long ago and is
> closed,
> > >> so
> > >> >> it
> > >> >> seems like it should have been cleaned up in the first place.
The
> > >> second
> > >> >> screenshot below shows a path from a GC root to the consumer.
It
> > looks
> > >> >> like
> > >> >> the _stateChangeListeners list of the head of the queue (which
has
> > been
> > >> >> sitting in Queue A for a long time) still contains this old
> > >> >> QueueConsumerImpl. We regularly create and close consumers to
our
> > >> queues,
> > >> >> so
> > >> >> it seems like these old consumers can then accumulate over time,
> > along
> > >> >> with
> > >> >> old consumers and old queue entries they reference. (Your fix
made
> > >> things
> > >> >> much better though when the old consumers accumulate.)
> > >> >> <
> > >> >>
> > >>
> >
> http://qpid.2158936.n2.nabble.com/file/n7633133/qpid_broker_heap_pathToQueueAConsumer.png
> > >> >> >
> > >> >>
> > >> >> So, the problem seems to be that even though a consumer has been
> > >> closed,
> > >> >> it
> > >> >> remains in a queue entry's _stateChangeListeners list. I see
> > >> >> QueueEntryImpl
> > >> >> has a removeStateChangeListener() method, but can't see where
its
> > >> caller
> > >> >> QueueEntryListener.stateChanged() gets called. So I'm not sure
how
> we
> > >> can
> > >> >> get into this situation. Would really appreciate your insight
on
> > this.
> > >> >>
> > >> >> Thanks!
> > >> >>
> > >> >> Helen
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> View this message in context:
> > >> >>
> > >>
> >
> http://qpid.2158936.n2.nabble.com/Unconsumed-messages-can-still-prevent-GC-of-consumed-messages-tp7633133.html
> > >> >> Sent from the Apache Qpid users mailing list archive at Nabble.com.
> > >> >>
> > >> >>
> ---------------------------------------------------------------------
> > >> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > >> >> For additional commands, e-mail: users-help@qpid.apache.org
> > >> >>
> > >> >>
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message