activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Green <james.mk.gr...@gmail.com>
Subject Re: DLQ, cause:null
Date Fri, 24 Apr 2015 14:10:49 GMT
I'm need to understand pre-fetch limit and receive time-out interaction.

We have four concurrent consumers in our route. Do each receive the
messages in batches of the pre-fetch limit?

At what point does the receive time-out start and end?

In our case each client performs a number of db queries then fires a new
message at the broker before the route is complete. Typically this may take
more than 1 second under load. A 10s time-out only makes sense if the
pre-fetching is not included but then that suggests client-calculated
time-outs communicated back to the broker which also makes no sense.

So, am clear we need to better understand what's under the hood here!

James

On 24 April 2015 at 14:39, Gary Tully <gary.tully@gmail.com> wrote:

> Tim, steady, I suggested it *may* be relevant :-)
> With camel and transactions - ie: spring dmlc, connection pools and
> cache levels - anything is possible w.r.t consumer/sessions/connection
> state, because there are so many variables in the mix.
>
> With activemq and prefetch, every consumer disconnect will result in
> redeliveries. The trick is figuring out whether
> the prefetched messages were actually delivered to the consumer so the
> delivery count can reflect the applications view
> of the world, that is not an exact science.
>
>
> On 24 April 2015 at 13:51, Tim Bain <tbain@alumni.duke.edu> wrote:
> > Gary,
> >
> > If I understood that JIRA correctly, the bug only occurs when the client
> > disconnects, which doesn't sound like what James is doing (nothing in his
> > description indicated to me that his client wasn't staying up and
> connected
> > the whole time), so it doesn't sound like your fix would resolve (nor
> > explain) his problem.  And although I'm all about workarounds when I know
> > there's a fix in a future version, I'm not sure that's the case here and
> I
> > don't want to give him a workaround at the expense of actually finding
> and
> > fixing a bug.
> >
> > The two things I know of that can cause message redelivery are 1) client
> > disconnection with queues and durable topic subscriptions and 2)
> unhandled
> > exceptions in the client message handler code.  James, might #2 be going
> on
> > here?  And Gary (or anyone else), are there any other possible causes of
> > redelivery that I don't know about?
> >
> > Tim
> > On Apr 24, 2015 4:59 AM, "Gary Tully" <gary.tully@gmail.com> wrote:
> >
> >> to avoid the redelivered messages getting sent to the DLQ, changing
> >> the default redelivery policy max from 6 to infinite will help.
> >>
> >> You can do this in the brokerurl passed to the jms connection factory,
> >> it may also make sense to reduce the prefetch if consumers come and go
> >> without consuming the prefetch, which seems to be the case.
> >>
> >>
> >>
> tcp://..:61616?jms.prefetchPolicy.all=100&jms.redeliveryPolicy.maximumRedeliveries=-1
> >>
> >> On 23 April 2015 at 17:14, James Green <james.mk.green@gmail.com>
> wrote:
> >> > Hi,
> >> >
> >> > We are not overriding so the defaults of 1s timeout on the receive()
> and
> >> > 1,000 prefetch are in play.
> >> >
> >> > We are updating the connection URI to set a much higher timeout.
> >> >
> >> > Interestingly, PHP sending to the very same broker via STOMP gets
> send()
> >> > fail with a 2 second timeout specified. With a 10 second timeout the
> >> > frequency of this is reduced.
> >> >
> >> > I have fired up the latest hawt.io jar and connected to this broker,
> >> > however the Health and Threads parts are entirely blank. The queues
> are
> >> all
> >> > visible yet "browse" of ActiveMQ.DLQ shows none of the 3,000+
> accumulated
> >> > messages. Wondering where to go next?
> >> >
> >> > Thanks,
> >> >
> >> > James
> >> >
> >> >
> >> > On 23 April 2015 at 13:35, Gary Tully <gary.tully@gmail.com> wrote:
> >> >
> >> >> what sort of timeout is on the receive(...) from spring dmlc, and
> what
> >> >> is the prefetch for that consumer. It appears that the message is
> >> >> getting dispatched but not consumed, the connection/consumer dies and
> >> >> the message is flagged as a redelivery. then the before delivery
> check
> >> >> on the delivery counter kicks the message to the dlq. So this must
be
> >> >> happening 6 times.
> >> >>
> >> >> I just pushed a tidy up of some of the redelivery semantics - there
> >> >> was a bug there that would cause the redelivery counter to increment
> >> >> in error... so that may be relevant[1].
> >> >> A short term solution would be to ensure infinite or a very large
> >> >> number of redeliveries, up from the default 6. That can be provided
> in
> >> >> the broker url.
> >> >>
> >> >> [1] https://issues.apache.org/jira/browse/AMQ-5735
> >> >>
> >> >> On 23 April 2015 at 13:08, James Green <james.mk.green@gmail.com>
> >> wrote:
> >> >> > We have a camel route consuming from ActiveMQ (5.10.0 with KahaDB)
> and
> >> >> > frequently get a DLQ entry without anything logged through our
> >> >> errorHandler.
> >> >> >
> >> >> > The only thing we have to go on is a dlqFailureCause header which
> >> says:
> >> >> >
> >> >> > java.lang.Throwable: Exceeded redelivery policy
> limit:RedeliveryPolicy
> >> >> > {destination = null, collisionAvoidanceFactor = 0.15,
> >> >> maximumRedeliveries =
> >> >> > 6, maximumRedeliveryDelay = -1, initialRedeliveryDelay = 1000,
> >> >> > useCollisionAvoidance = false, useExponentialBackOff = false,
> >> >> > backOffMultiplier = 5.0, redeliveryDelay = 1000}, cause:null
> >> >> >
> >> >> > These are happening apparently at random. The route is marked
> >> transacted,
> >> >> > and is backed by Spring Transactions itself backed by Narayana.
> >> >> >
> >> >> > Our debugging indicates that our route never receives the message
> from
> >> >> AMQ
> >> >> > prior to it hitting the DLQ. We have switched on DEBUG logging
for
> >> >> > org.apache.activemq but other than being swamped with even more
> logs
> >> >> we've
> >> >> > observed nothing notable.
> >> >> >
> >> >> > Any ideas where to go from here? Impossible to say which of the
> >> several
> >> >> > thousand messages per day will go this way so an attached debugger
> is
> >> out
> >> >> > of the question.
> >> >> >
> >> >> > Our log4j config fragment:
> >> >> >
> >> >> >         <Logger name="com" level="WARN"/>
> >> >> >         <Logger name="org" level="WARN"/>
> >> >> >         <Logger name="org.apache.camel" level="DEBUG"/>
> >> >> >         <Logger name="org.apache.activemq" level="DEBUG"/>
> >> >> >         <Logger name="org.springframework.orm.jpa" level="DEBUG"/>
> >> >> >         <Logger name="org.springframework.transaction"
> level="DEBUG"/>
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> > James
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message