camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kraythe ." <kray...@gmail.com>
Subject Re: Why do I get so many "external redeliveries" when using Camel with JMS topics?
Date Tue, 17 Jun 2014 14:59:18 GMT
A JMS topic will send a copy of each message to every topic. If not then
the system violates basic JMS spec. If the system is a mainstream system
like ActiveMQ, then that is not your problem. If you are trying to use a
queue as a topic it doesn't work that way.

A queue is like throwing candy into the middle of a class of elementary
kids. Each piece of candy will get consumed by only one kid but they will
scramble for the pieces of candy as fast as they can. However, if a kid
eats a piece of candy they can't regurgitate it and put it back into the
pile.

In your case if a route consumes a message on the queue, then it can't put
it back on the queue, the only way it gets back there is if the route fails
and rolls back.

Try adding a log line or a DLQ to the otherwise and see what happens. And
don't rollback in the otherwise. If you are using a queue, create a route
to sort the queue into 2 other queues based on the criteria and then
another route to process the queue where the relevant messages end up. i.e.
from(my
queue).choice().when(predicate).to("jms:queue:camel-processed-queue).otherwise().to("jms:queue:queue-processed-externally")


*Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
*Author of: Maintainable Java (Kindle
<http://www.amazon.com/Maintainable-Java-Robert-Simmons-Jr-ebook/dp/B00AKHI69K>)(iTunes
<https://itunes.apple.com/us/book/maintainable-java/id585666097?mt=11>)*
*LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
<http://www.linkedin.com/pub/robert-simmons/40/852/a39>*


On Tue, Jun 17, 2014 at 9:27 AM, Jeff Bischoff <jbischoff@wdtablesystems.com
> wrote:

> Robert, thanks so much for replying.
>
> > Without that ack, the external system thinks it is not consumed and in
> >good faith tries to redeliver.
>
> Okay, that seems like a good explanation for why I am getting external
> redeliveries!
>
> I guess the root of my problem is that it seems like my Camel consumer is
> competing with my non-Camel topic subscribers. To your question:
>
> > So the question you have to decide to solve your problem is, "What is the
> > route I want my messages to take if they are not in the when condition.
> >Do
> > they end up in a dead letter queue? Routed elsewhere? Logged to a file?
> >But
> > they have to end up somewhere.
>
> I have non-Camel endpoints that will pick up the messages that don't meet
> the filter requirements. Ideally, I would have an "otherwise" that does
> nothing with the message (but does send the ack that it has been
> processed). These particular messages do not need to be routed by Camel in
> my system.
>
> However, it seems like Camel is competing with my non-Camel endpoints to
> consumer topic messages. I would have thought each subscriber (Camel and
> the other subscriber) would get their own copy of the topic message, and
> that they would not compete with each other. It seems like the non-Camel
> consumer can only process the message if I do a "rollback" in Camel.
>
> I'm building JUnit tests to try to figure out what I'm doing wrong, but
> any further insight would be greatly appreciated.
>
>
> Best,
>
> Jeff
>
>
>
> On 6/17/14 9:16 AM, "kraythe ." <kraythe@gmail.com> wrote:
>
> >Good suggestions but you will need to check the JMS server config if you
> >are getting external redeliveries. Those happen when an external system to
> >the camel route performs a redelivery.
> >
> >As for your original route, there must always be a place for an exchange
> >to
> >go or it will stop routing. In the case of your when without otherwise,
> >what would be the path of the exchange that fails the when condition? It
> >is
> >probably returned to the external system in some manner and not acked as
> >being delivered so it is sent again. Without that ack, the external system
> >thinks it is not consumed and in good faith tries to redeliver.
> >
> >Furthermore when you rollback, you are telling the host JMS system that
> >you
> >did not consume the message, and it will try again of course.
> >
> >So the question you have to decide to solve your problem is, "What is the
> >route I want my messages to take if they are not in the when condition. Do
> >they end up in a dead letter queue? Routed elsewhere? Logged to a file?
> >But
> >they have to end up somewhere.
> >
> >*Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
> >*Author of: Hardcore Java (2003) and Maintainable Java (2012)*
> >*LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
> ><http://www.linkedin.com/pub/robert-simmons/40/852/a39>*
> >
> >
> >On Mon, Jun 16, 2014 at 5:56 PM, Minh Tran <darth.minhster@gmail.com>
> >wrote:
> >
> >> I have used topics in camel and they generally work the way I expect it.
> >>
> >> I would start by leaving the cache levels at the default and simplifying
> >> your route further. eg take out the choice and send it direct from
> >>topic to
> >> queue. Then add the choice but send it directly to the queue without the
> >> recipient list. Also enable tracing on the camel context so you can see
> >> what is happening to your message each step of the way.
> >>
> >> On 17/06/2014, at 6:28 AM, Jeff Bischoff <jbischoff@wdtablesystems.com>
> >> wrote:
> >>
> >> > Does nobody use Topics with Camel? They don't seem to work as
> >>expected.
> >> >
> >> > JB
> >> >
> >> > On 5/30/14 10:48 AM, "Jeff Bischoff" <jbischoff@wdtablesystems.com>
> >> wrote:
> >> >
> >> >> Correction:
> >> >>
> >> >> 1) Why would *CACHE_CONSUMER* on the ActiveMQComponent cause endless
> >> >> external
> >> >> redeliveries of a (filtered out) topic message, when with other cache
> >> >> settings (like CACHE_SESSION) this does not occur?
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 5/30/14 10:39 AM, "Jeff Bischoff" <jbischoff@wdtablesystems.com>
> >> wrote:
> >> >>
> >> >>> Maybe I need to narrow my questions:
> >> >>>
> >> >>> 1) Why would CACHE_SESSION on the ActiveMQComponent cause endless
> >> >>> external
> >> >>> redeliveries of a (filtered out) topic message, when with other
> >>cache
> >> >>> settings this does not occur?
> >> >>>
> >> >>> 2) Why would <rollback/> prevent the external redeliveries,
but
> >><stop/>
> >> >>> does not prevent them?
> >> >>>
> >> >>>
> >> >>> Did I provide enough info below to answer these questions? Topics
> >>just
> >> >>> aren't working in Camel the way that I would expect!
> >> >>>
> >> >>> Thanks,
> >> >>>
> >> >>> Jeff Bischoff
> >> >>> WDTS
> >> >>>
> >> >>> On 5/29/14 4:25 PM, "Jeff Bischoff" <jbischoff@wdtablesystems.com>
> >> wrote:
> >> >>>
> >> >>>> Hi all,
> >> >>>>
> >> >>>> Please forgive me if this is a basic question.
> >> >>>>
> >> >>>> Working with Camel and JMS Topics has seemed very finicky for
me so
> >> far.
> >> >>>> If I make slight configuration changes, I see the same message
> >> repeated
> >> >>>> endlessly. Using jconsole, I now know these messages are "external
> >> >>>> redeliveries", i.e. the same message is being sent from JMS
to
> >>Camel
> >> >>>> repeatedly without ever getting through. It happens in the
case of
> >> >>>> messages that I am intentionally not processing due to some
> >>criteria.
> >> >>>> What I don't understand is why these external redeliveries
keep
> >> >>>> occurring, when I simply want this message to be dropped.
> >> >>>>
> >> >>>> For example, if I use CACHE_SESSION or lower cache level, the
> >> following
> >> >>>> route works just fine for me:
> >> >>>>
> >> >>>>
> >> >>>>       <route id="topics-out">
> >> >>>>           <from uri="topic-jms:topic:projectx.>"/>
> >> >>>>           <log message="Picked up message with id: ${id},
> >> >>>> destinationType: ${header.DestinationEntityType}"
> >> loggingLevel="INFO"/>
> >> >>>>           <choice>
> >> >>>>               <when>
> >> >>>>                   <simple>${header.DestinationEntityType}
not in
> >> >>>> 'FOO,BAR'</simple>
> >> >>>>                   <log message="Detected new ${header.Name}
topic
> >> >>>> message on ${header.JMSDestination} with id: ${id},
> >>destinationType:
> >> >>>> ${header.DestinationEntityType}" loggingLevel="INFO"/>
> >> >>>>                   <recipientList ignoreInvalidEndpoints="false">
> >> >>>>
> >><simple>lc-jms:${header.JMSDestination}</simple>
> >> >>>>                   </recipientList>
> >> >>>>               </when>
> >> >>>>           </choice>
> >> >>>>       </route>
> >> >>>>
> >> >>>>
> >> >>>> However, if I switch to CACHE_CONSUMER and I have even one
message
> >> >>>> addressed to 'FOO' or 'BAR', I will see the endless external
> >> >>>> redeliveries. Note that I am using a SingleConnectionFactory
> >> >>>> specifically
> >> >>>> for this route.
> >> >>>>
> >> >>>>       <route id="topics-out">
> >> >>>>           <from uri="topic-jms:topic:projectx.>"/>
> >> >>>>           <log message="Picked up message with id: ${id},
> >> >>>> destinationType: ${header.DestinationEntityType}"
> >> loggingLevel="INFO"/>
> >> >>>>           <choice>
> >> >>>>               <when>
> >> >>>>                   <simple>${header.DestinationEntityType}
not in
> >> >>>> 'FOO,BAR'</simple>
> >> >>>>                   <log message="Detected new ${header.Name}
topic
> >> >>>> message on ${header.JMSDestination} with id: ${id},
> >>destinationType:
> >> >>>> ${header.DestinationEntityType}" loggingLevel="INFO"/>
> >> >>>>                   <recipientList ignoreInvalidEndpoints="false">
> >> >>>>
> >><simple>lc-jms:${header.JMSDestination}</simple>
> >> >>>>                   </recipientList>
> >> >>>>               </when>
> >> >>>>           </choice>
> >> >>>>       </route>
> >> >>>>
> >> >>>> Now, if I add an Otherwise clause with a Rollback, everything
works
> >> fine
> >> >>>> again (albeit with a handful of log messages due to the rollbacks):
> >> >>>>
> >> >>>>       <route id="topics-out">
> >> >>>>           <from uri="topic-jms:topic:projectx.>"/>
> >> >>>>           <log message="Picked up message with id: ${id},
> >> >>>> destinationType: ${header.DestinationEntityType}"
> >> loggingLevel="INFO"/>
> >> >>>>           <choice>
> >> >>>>               <when>
> >> >>>>                   <simple>${header.DestinationEntityType}
not in
> >> >>>> 'FOO,BAR'</simple>
> >> >>>>                   <log message="Detected new ${header.Name}
topic
> >> >>>> message on ${header.JMSDestination} with id: ${id},
> >>destinationType:
> >> >>>> ${header.DestinationEntityType}" loggingLevel="INFO"/>
> >> >>>>                   <recipientList ignoreInvalidEndpoints="false">
> >> >>>>
> >><simple>lc-jms:${header.JMSDestination}</simple>
> >> >>>>                   </recipientList>
> >> >>>>               </when>
> >> >>>>               <otherwise>
> >> >>>>               <rollback markRollbackOnly="true" />
> >> >>>>               </otherwise>
> >> >>>>           </choice>
> >> >>>>       </route>
> >> >>>>
> >> >>>> But if I change that Rollback to a Stop instead, I get the
endless
> >> >>>> external redeliveries again:
> >> >>>>
> >> >>>>       <route id="topics-out">
> >> >>>>           <from uri="topic-jms:topic:projectx.>"/>
> >> >>>>           <log message="Picked up message with id: ${id},
> >> >>>> destinationType: ${header.DestinationEntityType}"
> >> loggingLevel="INFO"/>
> >> >>>>           <choice>
> >> >>>>               <when>
> >> >>>>                   <simple>${header.DestinationEntityType}
not in
> >> >>>> 'FOO,BAR'</simple>
> >> >>>>                   <log message="Detected new ${header.Name}
topic
> >> >>>> message on ${header.JMSDestination} with id: ${id},
> >>destinationType:
> >> >>>> ${header.DestinationEntityType}" loggingLevel="INFO"/>
> >> >>>>                   <recipientList ignoreInvalidEndpoints="false">
> >> >>>>
> >><simple>lc-jms:${header.JMSDestination}</simple>
> >> >>>>                   </recipientList>
> >> >>>>               </when>
> >> >>>>               <otherwise>
> >> >>>>                <stop/>
> >> >>>>               </otherwise>
> >> >>>>           </choice>
> >> >>>>       </route>
> >> >>>>
> >> >>>> Why does rolling back prevent the unwanted message from being
> >> >>>> redelivered, while using "stop" causes the redelivery to occur?
> >> >>>> Intuitively, I would have thought it would work the opposite
way.
> >> >>>> Rolling
> >> >>>> back should put the item back on the "from" endpoint, allowing
it
> >>to
> >> be
> >> >>>> processed again. "Stop" should just drop the message, preventing
> >> further
> >> >>>> redelivery. What I'm actually seeing is the opposite of this.
Am I
> >> >>>> misunderstanding?
> >> >>>>
> >> >>>> I also get the same exact symptoms if I leave the cache level
at
> >> >>>> CACHE_SESSION, but I make the subscription durable. In that
case
> >>the
> >> >>>> rollbacks prevent the problem, but otherwise I get the eternal
> >> external
> >> >>>> redeliveries.
> >> >>>>
> >> >>>> Using a Filter element also produces the same results as using
the
> >> >>>> Choice
> >> >>>> element above:
> >> >>>>
> >> >>>>       <route id="topics-out">
> >> >>>>           <from
> >> >>>>
> >>uri="topic-jms:topic:projectx.>?durableSubscriptionName=testdurasub"/>
> >> >>>>           <log message="Picked up message with id: ${id},
> >> >>>> destinationType: ${header.DestinationEntityType}"
> >> loggingLevel="INFO"/>
> >> >>>>           <filter>
> >> >>>>               <simple>${header.DestinationEntityType}
not in
> >> >>>> 'FOO,BAR'</simple>
> >> >>>>               <recipientList ignoreInvalidEndpoints="false">
> >> >>>>                   <simple>lc-jms:${header.JMSDestination}</simple>
> >> >>>>               </recipientList>
> >> >>>>           </filter>
> >> >>>>       </route>
> >> >>>>
> >> >>>> When I look at the topic in JMS, I see the topic "projectx.>"
> >> appearing
> >> >>>> and disappearing in the list rapidly. I assume that's caused
by the
> >> >>>> Camel
> >> >>>> polling frequency. It seems a little disturbing to me; is that
> >>normal?
> >> >>>>
> >> >>>> Thanks so much for taking the time to help!
> >> >>>>
> >> >>>> Jeff Bischoff
> >> >>>> WDTS
> >> >>>
> >> >>
> >> >
> >> > ________________________________
> >> >
> >> > Please take note: This email, including attachments, contains
> >> information which may be confidential or legally privileged and is only
> >>for
> >> the use of the individual or entity to whom it is properly addressed.
> >>Any
> >> unauthorized review, use, disclosure, copying, or distribution is
> >> prohibited. If you have reason to believe that you have received this
> >> communication in error, or that it may be misaddressed or not intended
> >>for
> >> you, please destroy it and notify the sender immediately. Thank you.
> >>
> >>
>
> ________________________________
>
> Please take note: This email, including attachments, contains information
> which may be confidential or legally privileged and is only for the use of
> the individual or entity to whom it is properly addressed. Any unauthorized
> review, use, disclosure, copying, or distribution is prohibited. If you
> have reason to believe that you have received this communication in error,
> or that it may be misaddressed or not intended for you, please destroy it
> and notify the sender immediately. Thank you.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message