activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michiel Hendriks <michiel.hendr...@mp-objects.com>
Subject Re: Messages on queue but doesCursorHaveMessagesBuffered = false
Date Fri, 18 Sep 2015 07:12:13 GMT
Upgrading to JRE7 sadly did not fix the problem. It took a few days, but
system is now again in that weird state which makes little sense. I guess I
need to figure out a way to reproduce it.

On 8 September 2015 at 14:36, Tim Bain <tbain@alumni.duke.edu> wrote:

> Not that upgrading to JRE7 would be a bad thing, but I'd say upgrading from
> ActiveMQ 5.4.2 to 5.12 is probably the more important of the two.  Though
> as you said, it appears to still be a problem at least in 5.9, so there's
> no guarantee that upgrading ActiveMQ will fix the problem.
>
> My suggestion about the debugging port still stands, no matter which JRE
> and ActiveMQ versions you use.
> On Sep 8, 2015 2:06 AM, "Michiel Hendriks" <
> michiel.hendriks@mp-objects.com>
> wrote:
>
> > It's not a JVM GC, but executing the broker GC (which you can execute via
> > JMX) which flushes buffers, etc. And it's flushing of the cursor's
> prefetch
> > buffer that does the trick.
> >
> > The main annoying thing about this problem is that I cannot reproduce it.
> > And it only happens in our production environment once in a while. I fear
> > it might be a race condition in the JRE6 we use. So upgrading to JRE7
> might
> > solve the problem.
> >
> > On 3 September 2015 at 17:57, Tim Bain <tbain@alumni.duke.edu> wrote:
> >
> > > For me, the weirdest part about this is the fact that a full GC somehow
> > > fixes is.  GCs aren't supposed to do anything that would affect the
> > > application (garbage should always be dead, so there should be no
> effect
> > to
> > > anything that's still alive), so that's really strange.
> > >
> > > If you start ActiveMQ with a debug port enabled, you can always grab
> the
> > > 5.4.2 source code and attach a debugger once you get into this
> situation
> > > and see if stepping through the code gives any clues to how you got
> there
> > > (and whether the code that "should have" returned values from method
> > calls
> > > really is)...
> > >
> > > And I hope you're planning that upgrade for as soon as you can arrange
> > it.
> > >
> > > Tim
> > >
> > > On Wednesday, August 19, 2015, Michiel Hendriks <
> > > michiel.hendriks@mp-objects.com> wrote:
> > >
> > > > We've got a really weird problem. Some times we still have messages
> on
> > a
> > > > queue, but they are not being consumed by the active consumers.
> > > >
> > > > Via JMX the QueueSize is 1
> > > > doesCursorHaveMessagesBuffered return false
> > > > cursorSize returns 0
> > > > doesCursorHaveSpace returns true
> > > > I can browse the message
> > > >
> > > > Restarting consumers has no effect. Restarting ActiveMQ, or simply
> > > > performing a gc on the broker via JMX will correct it. Sending a
> > message
> > > to
> > > > the queue will push the current pending message to a consumer, but
> the
> > > new
> > > > message is then stuck.
> > > >
> > > > I've created and analyzed a memory dump. The batchList
> > > (OrderedPendingList)
> > > > of the QueueStorePrefetch contains a map with size 1, so
> > > > doesCursorHaveMessagesBuffered should have returned true rather than
> > > false.
> > > >
> > > > We have been unable to create a reproduction case yet, it just
> happens
> > at
> > > > some point at stays like that until a restart or explicit gc on the
> > > broker.
> > > > The queue has 20 consumers (prefetch=1), and a lot of messages do
> need
> > to
> > > > be redelivered.
> > > >
> > > > This is with ActiveMQ 5.4.2 running on java 1.6.0_36
> > > > (rhel-1.13.8.1.el5_11-x86_64). Yes, I know it's rather old (we don't
> > have
> > > > the luxury yet to upgrade). Looking at the code of a newer ActiveMQ I
> > do
> > > > not see differences in the path from doesCursorHaveMessagesBuffered
> to
> > > > OrderedPendingList.isEmpty
> > > >
> > > > Has anybody seen anything like this before?
> > > >
> > > > I did see a message from Donnell Alwyn on 04 Nov 2014 which looks
> quite
> > > > similar to this problem, but for AMQ 5.9. But there was no answer to
> > his
> > > > question.
> > > >
> > > > --
> > > > "If you dig a hole and it’s in the wrong place digging it deeper is
> not
> > > > going to help." – Seymour Chwast ​
> > > >
> > > > Michiel Hendriks, MSc
> > > > MP Objects - Supply Chain Systems
> > > >
> > > > ✉️ michiel.hendriks@mp-objects.com <javascript:;>
> > > > 🏠 www.mp-objects.com
> > > > ☎ +31 102900304
> > > >
> > >
> >
> >
> >
> > --
> > "If you dig a hole and it’s in the wrong place digging it deeper is not
> > going to help." – Seymour Chwast ​
> >
> > Michiel Hendriks, MSc
> > MP Objects - Supply Chain Systems
> >
> > ✉️ michiel.hendriks@mp-objects.com
> > 🏠 www.mp-objects.com
> > ☎ +31 102900304
> >
>



-- 
"If you dig a hole and it’s in the wrong place digging it deeper is not
going to help." – Seymour Chwast ​

Michiel Hendriks, MSc
MP Objects - Supply Chain Systems

✉️ michiel.hendriks@mp-objects.com
🏠 www.mp-objects.com
☎ +31 102900304

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message