activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin C." <mart...@gmx.at>
Subject Re: Messages stuck in queue which are not expired
Date Thu, 14 Apr 2016 12:09:59 GMT
Hi,

It's been quite some time now and I just wanted to follow up for
archiving purposes. Since we updated everything to 5.13, we don't see
the issue any more. I am convinced it had been caused by the 5.7
client library.

Best regards,
Martin

On Sat, Jan 9, 2016 at 6:24 AM, Tim Bain <tbain@alumni.duke.edu> wrote:
> Great, thanks for reporting back.
>
> Tim
>
> On Fri, Jan 8, 2016 at 1:04 PM, Martin C. <martinc@gmx.at> wrote:
>
>> Hello,
>>
>> sorry for the late response, missed the message. We are currently very
>> confident that the issue was triggered because one of the client
>> applications was still using a very old 5.7 library to connect to the
>> 5.12 broker. Once we updated the client library in that particular
>> application, things started to improve and we are currently testing if
>> upgrading everything across all clients and also the broker to 5.13
>> fixes this for good.
>>
>> Once done, I'll post our test results.
>>
>> Best regards,
>> Martin
>>
>>
>> On Sun, Dec 20, 2015 at 4:50 PM, Tim Bain <tbain@alumni.duke.edu> wrote:
>> > Martin, did you ever resolve this issue?
>> >
>> > If not, I'd recommend looking at the messages that expire to see if there
>> > is a pattern to them.
>> >
>> > Also, do you have a single broker, or a network of brokers?  If the
>> latter,
>> > what is your networkTTL set to?
>> >
>> > Tim
>> > On Dec 2, 2015 9:50 AM, "Martin Carpella" <martin.carpella@gmail.com>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> We've ran into some problems since we updated to Activemq 5.12.1. Our
>> >> most busy queue has stuck messages which also do NOT expire.
>> >> The queue has around 200 producers (each producer has it's own message
>> >> group, making sure messages of a producer do not overtake each other)
>> >> which send non-persistent messages with a timeout of 40 seconds. They
>> >> produce around 20-30 msgs / second. 5 cached consumers exist.
>> >>
>> >> Our problem is that all 5 consumers are consuming messages but some of
>> >> those messages are apparently not delivered. They get stuck in the
>> >> queue and stay there. They do not expire.
>> >> The only solution to "clear" the queue is to use a QueueBrowser and
>> >> inspect it. Once I connect with the QueueBrowser, all messages are
>> >> apparently moved to expiration. After that the processing works for a
>> >> couple of minutes until the messages start clogging up again.
>> >>
>> >> The consumers do not use any form of selector other than the JMS
>> >> message group. The operation on the server side is very lightweight
>> >> and the load on the server is low so i do not think that it's the
>> >> fault of the server for not processing the messages fast enough (and
>> >> they should at least time out after their expiration deadline is
>> >> reached).
>> >> The problem scales apparently with the amount of the producers /
>> >> produced messages. Systems with ~100 producers have much fewer stuck
>> >> messages.
>> >>
>> >> All our other queues use message groups as well but work as intended.
>> >> A maybe noticable difference is that the messages that get stuck are
>> >> non-persistent and have a TTL. We have some high-throughput queues
>> >> with non-expiring, non-persistent messages, which do not show those
>> >> symptoms.
>> >>
>> >> Good ideas on what could be the issues are very welcome! Thanks in
>> advance!
>> >>
>> >> Best regards,
>> >> Martin
>> >>
>>

Mime
View raw message