camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alistair Young <alistair.yo...@uhi.ac.uk>
Subject Re: Messages being lost from route
Date Fri, 30 Sep 2011 09:27:46 GMT
I can now reproduce this every time. Sending 100 messages in very quick succession to a camel
route. JMX:

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
      <from uri="activemq:topic:matrix"/>
      <to uri="activemq:topic:edirectory"/>
    </route>

EnqueueCount = 100
DispatchCount = 100
InFlightCount = 100
DequeueCount = 1

only 1 message ever gets through the route. That's 99 percent message loss, every time. This
happens on two servers with the same config.

This happens with the producer on the same machine as the broker as well as a remote broker.

Is there anything I should be looking at to see where the other 99 messages are? JMX says
they're in flight but all dead letter queues are empty and there are no errors anywhere.

Alistair

-- 
mov eax,1
mov ebx,0
int 80h




On 30 Sep 2011, at 09:12, Alistair Young wrote:

> getting somewhere.
> 
>    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>      <from uri="activemq:topic:matrix"/>
>      <transacted />
>      <process ref="matrixProcessor" />
>      <to uri="activemq:topic:edirectory"/>
>    </route>
> 
> - producer clock is 2mins ahead of broker clock, timestampplugin enabled on broker
> - no delay between messages at the producer = 62 out of 100 get through the route, consistently
> - 3sec delay at producer, around 88 - 97 get through. That's as good as it gets
> - 5sec delay at producer, 93 made it through the route
> - remove the route and send direct to topic in activemq, no delay, no message loss, consistently
> 
> so the messages are being lost in the route. No matter what the delay in sending messages,
some are always lost and vanish. There are no errors, nothing in any dead letter queue. They
simply vanish. They don't even make it as far as the <process ref="matrixProcessor" />.
> 
> sending from a ruby producer that blats them out far quicker than the java producer is
even worse. Only 1 - 3 ever get through the route. Removing the route and sending to the activemq
topic instead, all messages get through no matter how fast they come.
> 
> Alistair
> 
> -- 
> mov eax,1
> mov ebx,0
> int 80h
> 
> 
> 
> 
> On 29 Sep 2011, at 18:10, Alistair Young wrote:
> 
>> no connection pool. What was disturbing was the first message sent to the broker
after a restart and clean out of the data dir, disappeared. There's a similar route on the
broker that works fine. The only difference is the producer for the wonky route is on windows
and is up to 1min ahead of the broker's clock. Would have thought the timestamplugin would
take care of that though. Can see it changing the timestamp in the logs.
>> 
>> Alistair
>> 
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>> 
>> On 29 Sep 2011, at 17:31, Tim wrote:
>> 
>>> Well with only 17 you definitely aren't hitting any prefetch limits or
>>> anything like that.
>>> Are you using a connection pool?
>>> 
>>> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <alistair.young@uhi.ac.uk>wrote:
>>> 
>>>> I think this way madness lies.
>>>> 
>>>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
>>>> topicB enqueueCount = 12
>>>> 
>>>> so 17 came in, 12 made it through, of the 5 that went missing it claims to
>>>> have sent 3 to topicB but they never arrived and the last 2 just simply
>>>> vanished completely.
>>>> 
>>>> What on earth?
>>>> 
>>>> Alistair
>>>> 
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>> 
>>>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
>>>> 
>>>>> nup - cleaned out the data dir and restarted the broker. First message
in
>>>> vanished. Wasn't persisted. So something is fundamentally broken.
>>>>> 
>>>>> topicA inflightCount = dispatchCount = enqueueCount = 1
>>>>> topicB is completely empty
>>>>> 
>>>>> so the message wasn't persisted, wasn't processed, wasn't routed and
just
>>>> vanished from the broker.
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> --------------
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80
>>>>> 
>>>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
>>>>> 
>>>>>> route goes from topicA -> topicB, transacted.
>>>>>> topicA inflightCount = 96 and increases on each batch of incoming
>>>> messages
>>>>>> topicB dispatchCount = enqueueCount
>>>>>> 
>>>>>> wondering if the missing messages are connected to topicA inflightCount.
>>>> I noticed there are two consumers for topicB. The main consumer gets its
>>>> messages fine. Wonder if the second consumer is a durable topic consumer
and
>>>> therefore activemq is persisting its messages but it hasn't connected in
a
>>>> very long time. Would that cause the topic to get too big? i.e. messages
go
>>>> into the topic until the limit is reached. Main consumer pulls messages off
>>>> and messages are able to go onto topicB again. Before consumer pulls and
>>>> after limit reached, messages can't get from topicA -> topicB, hence the
>>>> topicA inflightCount not zero?
>>>>>> 
>>>>>> Alistair
>>>>>> 
>>>>>> --------------
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80
>>>>>> 
>>>>>> On 29 Sep 2011, at 12:17, Tim wrote:
>>>>>> 
>>>>>>> Sorry you might have tried this since I haven't been following
this
>>>> thread.
>>>>>>> But can you check your jmx console.
>>>>>>> In particular check 2 things.. the route to see if the number
of
>>>> exchanges
>>>>>>> match what you think and how if any exchanges failed.
>>>>>>> Also check the JMX console on activemq for the queue or topic
in
>>>> question
>>>>>>> and see how many were enqueued vs dispatched.
>>>>>>> Check your deadletter queue from there too
>>>>>>> 
>>>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>>>>>> <alistair.young@uhi.ac.uk>wrote:
>>>>>>> 
>>>>>>>> dunno - nothing works. Random messages are just vanishing
once they
>>>> reach
>>>>>>>> the broker. No trace, no logs, no dead letter queue. Just
vanishing.
>>>> I've
>>>>>>>> removed <transacted /> and <process> but it doesn't
help. The producer
>>>> is a
>>>>>>>> few secs behind the broker:
>>>>>>>> 
>>>>>>>> sent : 11:25:26
>>>>>>>> arrived : 11:24:57
>>>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57
GMT,
>>>> presumably
>>>>>>>> the timestampplugin doing this
>>>>>>>> message vanishes
>>>>>>>> 
>>>>>>>> but all messages display this clock behaviour and not all
vanish.
>>>>>>>> 
>>>>>>>> Alistair
>>>>>>>> 
>>>>>>>> --------------
>>>>>>>> mov eax,1
>>>>>>>> mov ebx,0
>>>>>>>> int 80
>>>>>>>> 
>>>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>>>>>> 
>>>>>>>>> just saw your info about transacted being before from
- will change
>>>> that
>>>>>>>> and monitor again.
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> 
>>>>>>>>> Alistair
>>>>>>>>> 
>>>>>>>>> --------------
>>>>>>>>> mov eax,1
>>>>>>>>> mov ebx,0
>>>>>>>>> int 80
>>>>>>>>> 
>>>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>>>>>> 
>>>>>>>>>> just noticed a batch of identical 5 messages, three
were missing and
>>>>>>>> another single message vanished. tracer logged nothing. No
errors,
>>>> dead
>>>>>>>> letter queue empty.
>>>>>>>>>> 
>>>>>>>>>> One thing that happens is another machine polls the
stats topic in
>>>>>>>> activemq every 2mins. Would that cause a problem? It asks
for stats on
>>>> the
>>>>>>>> matrix topic, which is part of the transacted route.
>>>>>>>>>> 
>>>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>>>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction
with name
>>>> [null]:
>>>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
>>>> vm://matrixBroker#285916
>>>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection
id:
>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961,
address:
>>>>>>>> vm://matrixBroker#285920
>>>>>>>>>> 29 September 2011 10:05:07 - Adding Connection :
ConnectionInfo
>>>>>>>> {commandId = 1, responseRequired = true, connectionId =
>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961,
clientId =
>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961,
userName =
>>>> null,
>>>>>>>> password = *****, brokerPath = null, brokerMasterConnector
= false,
>>>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>>>>>> 
>>>>>>>>>> Alistair
>>>>>>>>>> 
>>>>>>>>>> --------------
>>>>>>>>>> mov eax,1
>>>>>>>>>> mov ebx,0
>>>>>>>>>> int 80
>>>>>>>>>> 
>>>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>>>>>> 
>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>> it is after from - do you mean it should be before?
>>>>>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>>>>>> <transacted />
>>>>>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>>>>>> </route>
>>>>>>>>>>> 
>>>>>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>>>>>> 
>>>>>>>>>>> Alistair
>>>>>>>>>>> 
>>>>>>>>>>> --------------
>>>>>>>>>>> mov eax,1
>>>>>>>>>>> mov ebx,0
>>>>>>>>>>> int 80
>>>>>>>>>>> 
>>>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi
>>>>>>>>>>>> 
>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair
Young
>>>>>>>>>>>> <alistair.young@uhi.ac.uk> wrote:
>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>> no
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> timestamp plugin
>>>>>>>>>>>>> 
>>>>>>>>>>>>> using that
>>>>>>>>>>>>> 
>>>>>>>>>>>>> activemq 5.5.0
>>>>>>>>>>>>> camel 2.8.0
>>>>>>>>>>>>> spring 3.0.5
>>>>>>>>>>>>> 
>>>>>>>>>>>>> noticed sl4j errors on startup, fixed
that and now the tracer is
>>>>>>>> logging so hopefully I can see any errors.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <route id="matrix"
>>>> errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>>>>>> <transacted />
>>>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>>>>>> </route>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <bean id="jmsConnectionFactory"
>>>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>>>>>> depends-on="matrixBrokerID">
>>>>>>>>>>>>>       <property name="brokerURL"
>>>>>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <bean id="jmsTransactionManager"
>>>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>>>>>>       <property name="connectionFactory"
>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <bean id="activemq"
>>>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>>>>>>       <property name="connectionFactory"
>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>>       <property name="transacted"
value="true"/>
>>>>>>>>>>>>>       <property name="transactionManager"
>>>>>>>> ref="jmsTransactionManager"/>
>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>>>>>> <property name="redeliveryPolicy"
>>>>>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>>>>>> <property name="maximumRedeliveries"
value="10"/>
>>>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --------------
>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>> int 80
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen
wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>>> Make sure clocks between server/clients
is synced as much as
>>>>>>>> possible.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>>>>>> What version of AMQ and Camel are
you using?
>>>>>>>>>>>>>> And how have you configured the AMQ
broker, and the Camel
>>>> context?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM,
Taariq Levack <
>>>> taariql@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Where the logs go, if it's logged
at all, still depends on your
>>>>>>>> logger and
>>>>>>>>>>>>>>> how you configured it.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Here are links to how to enable
logging[1] and camel logging
>>>> FAQ[2]
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Taariq
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23
PM, Alistair Young <
>>>>>>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> which is the best trace method
to use? trace="true", or
>>>>>>>> camelTracer and
>>>>>>>>>>>>>>>> traceFormatter beans? and
where does the log end up? I've
>>>> tried
>>>>>>>> them all but
>>>>>>>>>>>>>>>> no log appears.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>>> int 80h
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08,
Marco Westermann wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I suggest enable tracing
to see exactly what happens in your
>>>>>>>> route.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Am 28.09.2011 13:01,
schrieb Alistair Young:
>>>>>>>>>>>>>>>>>> I now have a dead
letter channel which is empty after losing
>>>> 9
>>>>>>>> out of 10
>>>>>>>>>>>>>>>> messages. I also added a
logging handler which logged nothing.
>>>>>>>> Verified the
>>>>>>>>>>>>>>>> messages arrived at the broker,
then they just vanished.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>> -----------------
>>>>>>>>>>>> FuseSource
>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>> 
> 


Mime
View raw message