camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Lost Messages in DeadLetterChannel.
Date Tue, 08 Sep 2009 15:03:11 GMT
I have added a TODO in the camel code to take a look at when time
permits later about the handled property.


On Tue, Sep 8, 2009 at 9:07 AM, Claus Ibsen<claus.ibsen@gmail.com> wrote:
> On Mon, Sep 7, 2009 at 9:56 PM, thabach<bach@evolootion.ch> wrote:
>>
>> Hi Claus
>>
>> I guess I found the root of the problem now.
>>
>> It is in my first segment's recovery logic. All my producer (the
>> CustomDirectProducer) does, in recovering from an IllegalStateException
>> thrown by a seda queue running full, is the following:
>>
>> exchange.setException(null);
>>
>> As I understand now, this seems not being sufficient, as the recovered
>> exchanges on consumption from the seda queues in the second segment, to my
>> surprise, do have the Exchange.FAILURE_HANDLED property set to true. The
>> property happens to be originally set by the DefaultErrorHandler in the
>> first segment on the 'queue full' IllegalStateException propagation.
>>
>
> Yeah you need to do a bit more work to clear traces of exceptions. The
> code below does clear all traces.
>
>        exchange.setException(null);
>
>        exchange.setProperty(Exchange.ERRORHANDLER_HANDLED, null);
>        exchange.setProperty(Exchange.FAILURE_HANDLED, null);
>        exchange.setProperty(Exchange.EXCEPTION_CAUGHT, null);
>        exchange.getIn().removeHeader(Exchange.REDELIVERED);
>        exchange.getIn().removeHeader(Exchange.REDELIVERY_COUNTER);
>
> I think Camel should have a helper method for that you can invoke, such as:
>   ExchangeHelper.clearException(exchange);
>
>
>
>> This becomes fatal in the second segment of my routes, as the
>> DeadLetterChannel in its isDone()-method considers the failed exchange as
>> being handled already and does not forward the Exchange to its deadletter
>> queue and the messages are 'lost'.
>>
>> I am left with two questions then :) , these are:
>>
>> - Shouldn't the DefaultErrorHandler as described
>> http://camel.apache.org/defaulterrorhandler.html here  refrain from marking
>> the exchange as being handled, i.e. leave it as unhandled ?
>>
>> - How do I properly recover an exchange from failure without creating a new
>> Exchange ? Is exchange.setException(null); and an additional
>> exchange.getProperties().clear(); sufficient ?
>>
>> Thanks for all your help and for supporting me digging deep, Christian.
>>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> Hi
>>>
>>> You test case is kinda bogus as the route is triggered by a file
>>> consumer, which splits the file and route from there.
>>>
>>> So if the caller in this case is the file consumer, how do you suppose
>>> the default error handler should "handle" this?
>>> It does this by rolling back the file and try the file again on next poll.
>>>
>>> So try with another kind of caller, for example try the dataset
>>> endpoint in camel that can be used for sending in loads of messages
>>> and verify they are routed as expected.
>>>
>>>
>>>               <route>
>>>                       <from uri="file://inputdir?noop=true" />
>>>                       <unmarshal>
>>>                               <string />
>>>                       </unmarshal>
>>>                       <split>
>>>                               <tokenize token="\n" />
>>>                               <to uri="customDirect:messageRouter"
/>
>>>                       </split>
>>>               </route>
>>>
>>> On Mon, Sep 7, 2009 at 1:15 PM, thabach<bach@evolootion.ch> wrote:
>>>>
>>>> I think we might suffer from some misunderstanding.
>>>>
>>>> - For the first segment, I do have a custom error handler configured to
>>>> redeliver failed messages to the seda queues, we are on the same page.
>>>>
>>>> - In the second segment, in the DeadLetterChannelTest, I do use a
>>>> DeadLetterChannel which does not, for some reason, deliver all messages.
>>>> Whereas in the ExceptionBarrierTest, I have another custom error handler
>>>> installed right after the seda queue, an ExceptionBarrier endpoint, which
>>>> simply forwards erroneous exchanges to the deadletter queue and leaves no
>>>> message behind. This second test does not use the DeadLetterChannel, but
>>>> rather shall illustrate that the general setup is not flawed.
>>>>
>>>> In both test cases the DeadLetterChannel and the ExceptionBarrier
>>>> respectively, are sitting right before the ExceptionThrower, which in
>>>> both
>>>> test cases deterministically throws 25 exceptional messages. Somehow in
>>>> the
>>>> DeadLetterChannelTest case, depending on the run only some 38 to 42
>>>> messages
>>>> out of 45 make it to the deadletter queue.
>>>>
>>>> Christian.
>>>>
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> On Mon, Sep 7, 2009 at 10:28 AM, thabach<bach@evolootion.ch> wrote:
>>>>>>
>>>>>> Hi Claus
>>>>>>
>>>>>> thanks for having a look, I appreciate.
>>>>>>
>>>>>> I don't think it is a misconfiguration though. On the first segment
to
>>>>>> the
>>>>>> seda queues I do want the default error handler to return the
>>>>>> 'exception'al-exchanges (i.e. the exchanges suffering from a 'Queue
>>>>>> full'
>>>>>> exception), to the producer. Theses messages are not supposed to
end up
>>>>>> in
>>>>>> the deadletter queue, but to be recovered by the producer
>>>>>> (CustomDirectProducer in the original unit tests, posted) and resent
to
>>>>>> the
>>>>>> seda queues.
>>>>>>
>>>>>
>>>>> Ah okay that is fine. Then we are on same page now. If you do *not*
>>>>> use the DeadLetterChannel then its your own responsible for handling
>>>>> errors.
>>>>> And whether you build you own custom component to do redelivery is your
>>>>> choice.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> With these 'blocking' producer semantics installed, all messages
make
>>>>>> it
>>>>>> to
>>>>>> the second segment (i.e. the routes consuming from the seda queues).
In
>>>>>> the
>>>>>> original unit tests the ExceptionThrower always throws on the expected
>>>>>> 25
>>>>>> messages, but not all of these 'exceptional' messages make it to
the
>>>>>> deadletter queue when using a DeadLetterChannel
>>>>>> (DeadLetterChannelTest),
>>>>>> whereas in the ExceptionBarrierTest - using our custom ExceptionBarrier
>>>>>> component - they do.
>>>>>>
>>>>>> I am still under the impression of losing messages and don't get
me
>>>>>> wrong, I
>>>>>> am happy to be told that it was just me (and my configuration) :)
.
>>>>>>
>>>>>
>>>>> No Camel does not lose messages. Its your own choice to do redelivery
>>>>> and error handling in your own custom component.
>>>>>
>>>>>
>>>>>> Cheers, Christian.
>>>>>>
>>>>>>
>>>>>> Claus Ibsen-2 wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I had a look at it.
>>>>>>> There are *no* issues in Camel. DeadLetterChannel does *not*
loose
>>>>>>> messages.
>>>>>>>
>>>>>>> The problem Christian has is that he has not properly configured
error
>>>>>>> handling in Camel.
>>>>>>> He have mixed error handlers
>>>>>>> - default error handler
>>>>>>> - dead letter channel error handler
>>>>>>>
>>>>>>> And since when he sends to the SEDA queue from a producer that
are
>>>>>>> handled by the default error handler,
>>>>>>> then his message is *not* moved to a dead letter queue, but returned
>>>>>>> back to the caller as an exception. (as it used the default error
>>>>>>> handler)
>>>>>>>
>>>>>>> Configuring Camel to use dead letter channel error handler for
the
>>>>>>> entire routes remedy Christians problem.
>>>>>>>
>>>>>>> I have added an unit test to Camel at:
>>>>>>> http://svn.apache.org/viewvc?rev=811997&view=rev
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 4, 2009 at 9:04 AM, Bach
>>>>>>> Christian<Christian.Bach@six-group.com> wrote:
>>>>>>>> Heya Camel Riders
>>>>>>>>
>>>>>>>> I stumbled upon a problem on having DeadLetterChannel doing
the error
>>>>>>>> handling in an staged architecture (all in-VM, non-distributed).
Our
>>>>>>>> batch processing application (uses Camel 2.0 and) is losing
messages.
>>>>>>>>
>>>>>>>> It seems to only happen under peak loads, when SEDA queues
become
>>>>>>>> full
>>>>>>>> and start to throw IllegalStateExceptions on performing the
add()s to
>>>>>>>> their internal BlockingQueues. Our producer recovers those
exchanges
>>>>>>>> and
>>>>>>>> yield()s, before re-sending the Exchanges to the SEDA endpoint.
>>>>>>>>
>>>>>>>> I have put together two unit tests that reproducibly illustrate
the
>>>>>>>> problem and the absence thereof. Both unit tests are similar
expect
>>>>>>>> for
>>>>>>>> the dead letter queue exception handling implementation used.
Both
>>>>>>>> have
>>>>>>>> the SEDA queues, facing stages, configured to size=1 in order
to
>>>>>>>> emulate
>>>>>>>> peak load semantics.
>>>>>>>>
>>>>>>>> The first test case (DeadLetterChannelTest) is configured
to use
>>>>>>>> Camel's
>>>>>>>> DeadLetterChannel and reproducibly loses messages, depending
on the
>>>>>>>> run
>>>>>>>> between 5 to 15 (out of 45) on my machine.
>>>>>>>>
>>>>>>>> The second test case (ExceptionBarrierTest) is configured
to use a
>>>>>>>> custom ExceptionBarrier endpoint in place of DeadLetterChannel.
The
>>>>>>>> ExceptionBarrierComponent is derived from a Camel DirectComponent
>>>>>>>> with
>>>>>>>> some custom code to handle Exchanges suffering from exceptions
and
>>>>>>>> forwarding the erroneous exchanges to the dead letter queue.
With the
>>>>>>>> ExceptionBarrier, which takes in its simplistic implementation
>>>>>>>> advantage
>>>>>>>> of our specific routes to some extent, no messages are lost.
>>>>>>>>
>>>>>>>> The unit test are attached and I'd be most interested in
learning
>>>>>>>> what
>>>>>>>> is going wrong.
>>>>>>>>
>>>>>>>> Cheers, Christian.
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> Apache Camel Committer
>>>>>>>
>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25326799.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25328848.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context: http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p25335523.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>



-- 
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Mime
View raw message