camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From thabach <b...@evolootion.ch>
Subject Re: Lost Messages in DeadLetterChannel.
Date Mon, 07 Sep 2009 19:56:28 GMT

Hi Claus

I guess I found the root of the problem now.

It is in my first segment's recovery logic. All my producer (the
CustomDirectProducer) does, in recovering from an IllegalStateException
thrown by a seda queue running full, is the following: 

exchange.setException(null);

As I understand now, this seems not being sufficient, as the recovered
exchanges on consumption from the seda queues in the second segment, to my
surprise, do have the Exchange.FAILURE_HANDLED property set to true. The
property happens to be originally set by the DefaultErrorHandler in the
first segment on the 'queue full' IllegalStateException propagation.

This becomes fatal in the second segment of my routes, as the
DeadLetterChannel in its isDone()-method considers the failed exchange as
being handled already and does not forward the Exchange to its deadletter
queue and the messages are 'lost'.

I am left with two questions then :) , these are:

- Shouldn't the DefaultErrorHandler as described 
http://camel.apache.org/defaulterrorhandler.html here  refrain from marking
the exchange as being handled, i.e. leave it as unhandled ?

- How do I properly recover an exchange from failure without creating a new
Exchange ? Is exchange.setException(null); and an additional
exchange.getProperties().clear(); sufficient ?

Thanks for all your help and for supporting me digging deep, Christian.

 

Claus Ibsen-2 wrote:
> 
> Hi
> 
> You test case is kinda bogus as the route is triggered by a file
> consumer, which splits the file and route from there.
> 
> So if the caller in this case is the file consumer, how do you suppose
> the default error handler should "handle" this?
> It does this by rolling back the file and try the file again on next poll.
> 
> So try with another kind of caller, for example try the dataset
> endpoint in camel that can be used for sending in loads of messages
> and verify they are routed as expected.
> 
> 
> 		<route>
> 			<from uri="file://inputdir?noop=true" />
> 			<unmarshal>
> 				<string />
> 			</unmarshal>
> 			<split>
> 				<tokenize token="\n" />
> 				<to uri="customDirect:messageRouter" />
> 			</split>
> 		</route>
> 
> On Mon, Sep 7, 2009 at 1:15 PM, thabach<bach@evolootion.ch> wrote:
>>
>> I think we might suffer from some misunderstanding.
>>
>> - For the first segment, I do have a custom error handler configured to
>> redeliver failed messages to the seda queues, we are on the same page.
>>
>> - In the second segment, in the DeadLetterChannelTest, I do use a
>> DeadLetterChannel which does not, for some reason, deliver all messages.
>> Whereas in the ExceptionBarrierTest, I have another custom error handler
>> installed right after the seda queue, an ExceptionBarrier endpoint, which
>> simply forwards erroneous exchanges to the deadletter queue and leaves no
>> message behind. This second test does not use the DeadLetterChannel, but
>> rather shall illustrate that the general setup is not flawed.
>>
>> In both test cases the DeadLetterChannel and the ExceptionBarrier
>> respectively, are sitting right before the ExceptionThrower, which in
>> both
>> test cases deterministically throws 25 exceptional messages. Somehow in
>> the
>> DeadLetterChannelTest case, depending on the run only some 38 to 42
>> messages
>> out of 45 make it to the deadletter queue.
>>
>> Christian.
>>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Mon, Sep 7, 2009 at 10:28 AM, thabach<bach@evolootion.ch> wrote:
>>>>
>>>> Hi Claus
>>>>
>>>> thanks for having a look, I appreciate.
>>>>
>>>> I don't think it is a misconfiguration though. On the first segment to
>>>> the
>>>> seda queues I do want the default error handler to return the
>>>> 'exception'al-exchanges (i.e. the exchanges suffering from a 'Queue
>>>> full'
>>>> exception), to the producer. Theses messages are not supposed to end up
>>>> in
>>>> the deadletter queue, but to be recovered by the producer
>>>> (CustomDirectProducer in the original unit tests, posted) and resent to
>>>> the
>>>> seda queues.
>>>>
>>>
>>> Ah okay that is fine. Then we are on same page now. If you do *not*
>>> use the DeadLetterChannel then its your own responsible for handling
>>> errors.
>>> And whether you build you own custom component to do redelivery is your
>>> choice.
>>>
>>>
>>>
>>>
>>>> With these 'blocking' producer semantics installed, all messages make
>>>> it
>>>> to
>>>> the second segment (i.e. the routes consuming from the seda queues). In
>>>> the
>>>> original unit tests the ExceptionThrower always throws on the expected
>>>> 25
>>>> messages, but not all of these 'exceptional' messages make it to the
>>>> deadletter queue when using a DeadLetterChannel
>>>> (DeadLetterChannelTest),
>>>> whereas in the ExceptionBarrierTest - using our custom ExceptionBarrier
>>>> component - they do.
>>>>
>>>> I am still under the impression of losing messages and don't get me
>>>> wrong, I
>>>> am happy to be told that it was just me (and my configuration) :) .
>>>>
>>>
>>> No Camel does not lose messages. Its your own choice to do redelivery
>>> and error handling in your own custom component.
>>>
>>>
>>>> Cheers, Christian.
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> I had a look at it.
>>>>> There are *no* issues in Camel. DeadLetterChannel does *not* loose
>>>>> messages.
>>>>>
>>>>> The problem Christian has is that he has not properly configured error
>>>>> handling in Camel.
>>>>> He have mixed error handlers
>>>>> - default error handler
>>>>> - dead letter channel error handler
>>>>>
>>>>> And since when he sends to the SEDA queue from a producer that are
>>>>> handled by the default error handler,
>>>>> then his message is *not* moved to a dead letter queue, but returned
>>>>> back to the caller as an exception. (as it used the default error
>>>>> handler)
>>>>>
>>>>> Configuring Camel to use dead letter channel error handler for the
>>>>> entire routes remedy Christians problem.
>>>>>
>>>>> I have added an unit test to Camel at:
>>>>> http://svn.apache.org/viewvc?rev=811997&view=rev
>>>>>
>>>>>
>>>>> On Fri, Sep 4, 2009 at 9:04 AM, Bach
>>>>> Christian<Christian.Bach@six-group.com> wrote:
>>>>>> Heya Camel Riders
>>>>>>
>>>>>> I stumbled upon a problem on having DeadLetterChannel doing the error
>>>>>> handling in an staged architecture (all in-VM, non-distributed).
Our
>>>>>> batch processing application (uses Camel 2.0 and) is losing messages.
>>>>>>
>>>>>> It seems to only happen under peak loads, when SEDA queues become
>>>>>> full
>>>>>> and start to throw IllegalStateExceptions on performing the add()s
to
>>>>>> their internal BlockingQueues. Our producer recovers those exchanges
>>>>>> and
>>>>>> yield()s, before re-sending the Exchanges to the SEDA endpoint.
>>>>>>
>>>>>> I have put together two unit tests that reproducibly illustrate the
>>>>>> problem and the absence thereof. Both unit tests are similar expect
>>>>>> for
>>>>>> the dead letter queue exception handling implementation used. Both
>>>>>> have
>>>>>> the SEDA queues, facing stages, configured to size=1 in order to
>>>>>> emulate
>>>>>> peak load semantics.
>>>>>>
>>>>>> The first test case (DeadLetterChannelTest) is configured to use
>>>>>> Camel's
>>>>>> DeadLetterChannel and reproducibly loses messages, depending on the
>>>>>> run
>>>>>> between 5 to 15 (out of 45) on my machine.
>>>>>>
>>>>>> The second test case (ExceptionBarrierTest) is configured to use
a
>>>>>> custom ExceptionBarrier endpoint in place of DeadLetterChannel. The
>>>>>> ExceptionBarrierComponent is derived from a Camel DirectComponent
>>>>>> with
>>>>>> some custom code to handle Exchanges suffering from exceptions and
>>>>>> forwarding the erroneous exchanges to the dead letter queue. With
the
>>>>>> ExceptionBarrier, which takes in its simplistic implementation
>>>>>> advantage
>>>>>> of our specific routes to some extent, no messages are lost.
>>>>>>
>>>>>> The unit test are attached and I'd be most interested in learning
>>>>>> what
>>>>>> is going wrong.
>>>>>>
>>>>>> Cheers, Christian.
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25326799.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25328848.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> Claus Ibsen
> Apache Camel Committer
> 
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> 
> 

-- 
View this message in context: http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25335523.html
Sent from the Camel - Users mailing list archive at Nabble.com.


Mime
View raw message