activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: DLQ, cause:null
Date Fri, 24 Apr 2015 13:39:08 GMT
Tim, steady, I suggested it *may* be relevant :-)
With camel and transactions - ie: spring dmlc, connection pools and
cache levels - anything is possible w.r.t consumer/sessions/connection
state, because there are so many variables in the mix.

With activemq and prefetch, every consumer disconnect will result in
redeliveries. The trick is figuring out whether
the prefetched messages were actually delivered to the consumer so the
delivery count can reflect the applications view
of the world, that is not an exact science.


On 24 April 2015 at 13:51, Tim Bain <tbain@alumni.duke.edu> wrote:
> Gary,
>
> If I understood that JIRA correctly, the bug only occurs when the client
> disconnects, which doesn't sound like what James is doing (nothing in his
> description indicated to me that his client wasn't staying up and connected
> the whole time), so it doesn't sound like your fix would resolve (nor
> explain) his problem.  And although I'm all about workarounds when I know
> there's a fix in a future version, I'm not sure that's the case here and I
> don't want to give him a workaround at the expense of actually finding and
> fixing a bug.
>
> The two things I know of that can cause message redelivery are 1) client
> disconnection with queues and durable topic subscriptions and 2) unhandled
> exceptions in the client message handler code.  James, might #2 be going on
> here?  And Gary (or anyone else), are there any other possible causes of
> redelivery that I don't know about?
>
> Tim
> On Apr 24, 2015 4:59 AM, "Gary Tully" <gary.tully@gmail.com> wrote:
>
>> to avoid the redelivered messages getting sent to the DLQ, changing
>> the default redelivery policy max from 6 to infinite will help.
>>
>> You can do this in the brokerurl passed to the jms connection factory,
>> it may also make sense to reduce the prefetch if consumers come and go
>> without consuming the prefetch, which seems to be the case.
>>
>>
>> tcp://..:61616?jms.prefetchPolicy.all=100&jms.redeliveryPolicy.maximumRedeliveries=-1
>>
>> On 23 April 2015 at 17:14, James Green <james.mk.green@gmail.com> wrote:
>> > Hi,
>> >
>> > We are not overriding so the defaults of 1s timeout on the receive() and
>> > 1,000 prefetch are in play.
>> >
>> > We are updating the connection URI to set a much higher timeout.
>> >
>> > Interestingly, PHP sending to the very same broker via STOMP gets send()
>> > fail with a 2 second timeout specified. With a 10 second timeout the
>> > frequency of this is reduced.
>> >
>> > I have fired up the latest hawt.io jar and connected to this broker,
>> > however the Health and Threads parts are entirely blank. The queues are
>> all
>> > visible yet "browse" of ActiveMQ.DLQ shows none of the 3,000+ accumulated
>> > messages. Wondering where to go next?
>> >
>> > Thanks,
>> >
>> > James
>> >
>> >
>> > On 23 April 2015 at 13:35, Gary Tully <gary.tully@gmail.com> wrote:
>> >
>> >> what sort of timeout is on the receive(...) from spring dmlc, and what
>> >> is the prefetch for that consumer. It appears that the message is
>> >> getting dispatched but not consumed, the connection/consumer dies and
>> >> the message is flagged as a redelivery. then the before delivery check
>> >> on the delivery counter kicks the message to the dlq. So this must be
>> >> happening 6 times.
>> >>
>> >> I just pushed a tidy up of some of the redelivery semantics - there
>> >> was a bug there that would cause the redelivery counter to increment
>> >> in error... so that may be relevant[1].
>> >> A short term solution would be to ensure infinite or a very large
>> >> number of redeliveries, up from the default 6. That can be provided in
>> >> the broker url.
>> >>
>> >> [1] https://issues.apache.org/jira/browse/AMQ-5735
>> >>
>> >> On 23 April 2015 at 13:08, James Green <james.mk.green@gmail.com>
>> wrote:
>> >> > We have a camel route consuming from ActiveMQ (5.10.0 with KahaDB)
and
>> >> > frequently get a DLQ entry without anything logged through our
>> >> errorHandler.
>> >> >
>> >> > The only thing we have to go on is a dlqFailureCause header which
>> says:
>> >> >
>> >> > java.lang.Throwable: Exceeded redelivery policy limit:RedeliveryPolicy
>> >> > {destination = null, collisionAvoidanceFactor = 0.15,
>> >> maximumRedeliveries =
>> >> > 6, maximumRedeliveryDelay = -1, initialRedeliveryDelay = 1000,
>> >> > useCollisionAvoidance = false, useExponentialBackOff = false,
>> >> > backOffMultiplier = 5.0, redeliveryDelay = 1000}, cause:null
>> >> >
>> >> > These are happening apparently at random. The route is marked
>> transacted,
>> >> > and is backed by Spring Transactions itself backed by Narayana.
>> >> >
>> >> > Our debugging indicates that our route never receives the message from
>> >> AMQ
>> >> > prior to it hitting the DLQ. We have switched on DEBUG logging for
>> >> > org.apache.activemq but other than being swamped with even more logs
>> >> we've
>> >> > observed nothing notable.
>> >> >
>> >> > Any ideas where to go from here? Impossible to say which of the
>> several
>> >> > thousand messages per day will go this way so an attached debugger
is
>> out
>> >> > of the question.
>> >> >
>> >> > Our log4j config fragment:
>> >> >
>> >> >         <Logger name="com" level="WARN"/>
>> >> >         <Logger name="org" level="WARN"/>
>> >> >         <Logger name="org.apache.camel" level="DEBUG"/>
>> >> >         <Logger name="org.apache.activemq" level="DEBUG"/>
>> >> >         <Logger name="org.springframework.orm.jpa" level="DEBUG"/>
>> >> >         <Logger name="org.springframework.transaction" level="DEBUG"/>
>> >> >
>> >> > Thanks,
>> >> >
>> >> > James
>> >>
>>

Mime
View raw message