Return-Path: Delivered-To: apmail-camel-users-archive@www.apache.org Received: (qmail 81740 invoked from network); 8 Sep 2009 15:04:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 8 Sep 2009 15:04:04 -0000 Received: (qmail 44749 invoked by uid 500); 8 Sep 2009 15:04:04 -0000 Delivered-To: apmail-camel-users-archive@camel.apache.org Received: (qmail 44719 invoked by uid 500); 8 Sep 2009 15:04:04 -0000 Mailing-List: contact users-help@camel.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@camel.apache.org Delivered-To: mailing list users@camel.apache.org Received: (qmail 44709 invoked by uid 99); 8 Sep 2009 15:04:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Sep 2009 15:04:04 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of claus.ibsen@gmail.com designates 209.85.218.216 as permitted sender) Received: from [209.85.218.216] (HELO mail-bw0-f216.google.com) (209.85.218.216) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Sep 2009 15:03:53 +0000 Received: by bwz12 with SMTP id 12so67698bwz.20 for ; Tue, 08 Sep 2009 08:03:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:content-type :content-transfer-encoding; bh=GJaUrxnza5XD3ko5vvhOyGCnMlFHMrO8lzgKlZMm/mM=; b=GWftejPJSCPC0N8+fmooL3v2g7Mp0PIs+rDvjbfgShshiUHZAFDo9Rin5zwJxD/tyg uONY6mAr4lgsMV6XXoWAYa1Vdhvt7EjGrvJdq4MSKeWNgrCH4NZ+3dQRs4M7M0XwgJqg 4C+G/Y/d+zOlhdaBt+M6BdKDSRUJd79Hl7tZo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; b=Z7+HBWgbG9TTiyyCeo0gbk6YGgNsLiiH4bXNBdXve65HyRw7eg+gNmO4kvx49/iObo QL/81vcs6rbmgI+MmeUQfh9a/C4Vec5J2UY/lwqVs76YtAzGRUuy14LePyCmMa1xrB7o Wubw4r8jcXW1d0nUCW5ja5cxAEJvPEpO+wJms= MIME-Version: 1.0 Received: by 10.204.156.28 with SMTP id u28mr13326929bkw.144.1252422211123; Tue, 08 Sep 2009 08:03:31 -0700 (PDT) In-Reply-To: <5380c69c0909080007o36ac07desc68fb9783d6516c@mail.gmail.com> References: <5380c69c0909062213y15816d2cl779a8dbdf668960c@mail.gmail.com> <25326799.post@talk.nabble.com> <5380c69c0909070152x16daf280g17eeec76cfc71bad@mail.gmail.com> <25328848.post@talk.nabble.com> <5380c69c0909070436y1e451b39x4ee700e0dfd40488@mail.gmail.com> <25335523.post@talk.nabble.com> <5380c69c0909080007o36ac07desc68fb9783d6516c@mail.gmail.com> From: Claus Ibsen Date: Tue, 8 Sep 2009 17:03:11 +0200 Message-ID: <5380c69c0909080803u1612e1e7qcdb1f716d627788c@mail.gmail.com> Subject: Re: Lost Messages in DeadLetterChannel. To: users@camel.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org I have added a TODO in the camel code to take a look at when time permits later about the handled property. On Tue, Sep 8, 2009 at 9:07 AM, Claus Ibsen wrote: > On Mon, Sep 7, 2009 at 9:56 PM, thabach wrote: >> >> Hi Claus >> >> I guess I found the root of the problem now. >> >> It is in my first segment's recovery logic. All my producer (the >> CustomDirectProducer) does, in recovering from an IllegalStateException >> thrown by a seda queue running full, is the following: >> >> exchange.setException(null); >> >> As I understand now, this seems not being sufficient, as the recovered >> exchanges on consumption from the seda queues in the second segment, to = my >> surprise, do have the Exchange.FAILURE_HANDLED property set to true. The >> property happens to be originally set by the DefaultErrorHandler in the >> first segment on the 'queue full' IllegalStateException propagation. >> > > Yeah you need to do a bit more work to clear traces of exceptions. The > code below does clear all traces. > > =A0 =A0 =A0 =A0exchange.setException(null); > > =A0 =A0 =A0 =A0exchange.setProperty(Exchange.ERRORHANDLER_HANDLED, null); > =A0 =A0 =A0 =A0exchange.setProperty(Exchange.FAILURE_HANDLED, null); > =A0 =A0 =A0 =A0exchange.setProperty(Exchange.EXCEPTION_CAUGHT, null); > =A0 =A0 =A0 =A0exchange.getIn().removeHeader(Exchange.REDELIVERED); > =A0 =A0 =A0 =A0exchange.getIn().removeHeader(Exchange.REDELIVERY_COUNTER)= ; > > I think Camel should have a helper method for that you can invoke, such a= s: > =A0 ExchangeHelper.clearException(exchange); > > > >> This becomes fatal in the second segment of my routes, as the >> DeadLetterChannel in its isDone()-method considers the failed exchange a= s >> being handled already and does not forward the Exchange to its deadlette= r >> queue and the messages are 'lost'. >> >> I am left with two questions then :) , these are: >> >> - Shouldn't the DefaultErrorHandler as described >> http://camel.apache.org/defaulterrorhandler.html here =A0refrain from ma= rking >> the exchange as being handled, i.e. leave it as unhandled ? >> >> - How do I properly recover an exchange from failure without creating a = new >> Exchange ? Is exchange.setException(null); and an additional >> exchange.getProperties().clear(); sufficient ? >> >> Thanks for all your help and for supporting me digging deep, Christian. >> >> >> >> Claus Ibsen-2 wrote: >>> >>> Hi >>> >>> You test case is kinda bogus as the route is triggered by a file >>> consumer, which splits the file and route from there. >>> >>> So if the caller in this case is the file consumer, how do you suppose >>> the default error handler should "handle" this? >>> It does this by rolling back the file and try the file again on next po= ll. >>> >>> So try with another kind of caller, for example try the dataset >>> endpoint in camel that can be used for sending in loads of messages >>> and verify they are routed as expected. >>> >>> >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 >>> >>> On Mon, Sep 7, 2009 at 1:15 PM, thabach wrote: >>>> >>>> I think we might suffer from some misunderstanding. >>>> >>>> - For the first segment, I do have a custom error handler configured t= o >>>> redeliver failed messages to the seda queues, we are on the same page. >>>> >>>> - In the second segment, in the DeadLetterChannelTest, I do use a >>>> DeadLetterChannel which does not, for some reason, deliver all message= s. >>>> Whereas in the ExceptionBarrierTest, I have another custom error handl= er >>>> installed right after the seda queue, an ExceptionBarrier endpoint, wh= ich >>>> simply forwards erroneous exchanges to the deadletter queue and leaves= no >>>> message behind. This second test does not use the DeadLetterChannel, b= ut >>>> rather shall illustrate that the general setup is not flawed. >>>> >>>> In both test cases the DeadLetterChannel and the ExceptionBarrier >>>> respectively, are sitting right before the ExceptionThrower, which in >>>> both >>>> test cases deterministically throws 25 exceptional messages. Somehow i= n >>>> the >>>> DeadLetterChannelTest case, depending on the run only some 38 to 42 >>>> messages >>>> out of 45 make it to the deadletter queue. >>>> >>>> Christian. >>>> >>>> >>>> >>>> Claus Ibsen-2 wrote: >>>>> >>>>> On Mon, Sep 7, 2009 at 10:28 AM, thabach wrote: >>>>>> >>>>>> Hi Claus >>>>>> >>>>>> thanks for having a look, I appreciate. >>>>>> >>>>>> I don't think it is a misconfiguration though. On the first segment = to >>>>>> the >>>>>> seda queues I do want the default error handler to return the >>>>>> 'exception'al-exchanges (i.e. the exchanges suffering from a 'Queue >>>>>> full' >>>>>> exception), to the producer. Theses messages are not supposed to end= up >>>>>> in >>>>>> the deadletter queue, but to be recovered by the producer >>>>>> (CustomDirectProducer in the original unit tests, posted) and resent= to >>>>>> the >>>>>> seda queues. >>>>>> >>>>> >>>>> Ah okay that is fine. Then we are on same page now. If you do *not* >>>>> use the DeadLetterChannel then its your own responsible for handling >>>>> errors. >>>>> And whether you build you own custom component to do redelivery is yo= ur >>>>> choice. >>>>> >>>>> >>>>> >>>>> >>>>>> With these 'blocking' producer semantics installed, all messages mak= e >>>>>> it >>>>>> to >>>>>> the second segment (i.e. the routes consuming from the seda queues).= In >>>>>> the >>>>>> original unit tests the ExceptionThrower always throws on the expect= ed >>>>>> 25 >>>>>> messages, but not all of these 'exceptional' messages make it to the >>>>>> deadletter queue when using a DeadLetterChannel >>>>>> (DeadLetterChannelTest), >>>>>> whereas in the ExceptionBarrierTest - using our custom ExceptionBarr= ier >>>>>> component - they do. >>>>>> >>>>>> I am still under the impression of losing messages and don't get me >>>>>> wrong, I >>>>>> am happy to be told that it was just me (and my configuration) :) . >>>>>> >>>>> >>>>> No Camel does not lose messages. Its your own choice to do redelivery >>>>> and error handling in your own custom component. >>>>> >>>>> >>>>>> Cheers, Christian. >>>>>> >>>>>> >>>>>> Claus Ibsen-2 wrote: >>>>>>> >>>>>>> Hi >>>>>>> >>>>>>> I had a look at it. >>>>>>> There are *no* issues in Camel. DeadLetterChannel does *not* loose >>>>>>> messages. >>>>>>> >>>>>>> The problem Christian has is that he has not properly configured er= ror >>>>>>> handling in Camel. >>>>>>> He have mixed error handlers >>>>>>> - default error handler >>>>>>> - dead letter channel error handler >>>>>>> >>>>>>> And since when he sends to the SEDA queue from a producer that are >>>>>>> handled by the default error handler, >>>>>>> then his message is *not* moved to a dead letter queue, but returne= d >>>>>>> back to the caller as an exception. (as it used the default error >>>>>>> handler) >>>>>>> >>>>>>> Configuring Camel to use dead letter channel error handler for the >>>>>>> entire routes remedy Christians problem. >>>>>>> >>>>>>> I have added an unit test to Camel at: >>>>>>> http://svn.apache.org/viewvc?rev=3D811997&view=3Drev >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 4, 2009 at 9:04 AM, Bach >>>>>>> Christian wrote: >>>>>>>> Heya Camel Riders >>>>>>>> >>>>>>>> I stumbled upon a problem on having DeadLetterChannel doing the er= ror >>>>>>>> handling in an staged architecture (all in-VM, non-distributed). O= ur >>>>>>>> batch processing application (uses Camel 2.0 and) is losing messag= es. >>>>>>>> >>>>>>>> It seems to only happen under peak loads, when SEDA queues become >>>>>>>> full >>>>>>>> and start to throw IllegalStateExceptions on performing the add()s= to >>>>>>>> their internal BlockingQueues. Our producer recovers those exchang= es >>>>>>>> and >>>>>>>> yield()s, before re-sending the Exchanges to the SEDA endpoint. >>>>>>>> >>>>>>>> I have put together two unit tests that reproducibly illustrate th= e >>>>>>>> problem and the absence thereof. Both unit tests are similar expec= t >>>>>>>> for >>>>>>>> the dead letter queue exception handling implementation used. Both >>>>>>>> have >>>>>>>> the SEDA queues, facing stages, configured to size=3D1 in order to >>>>>>>> emulate >>>>>>>> peak load semantics. >>>>>>>> >>>>>>>> The first test case (DeadLetterChannelTest) is configured to use >>>>>>>> Camel's >>>>>>>> DeadLetterChannel and reproducibly loses messages, depending on th= e >>>>>>>> run >>>>>>>> between 5 to 15 (out of 45) on my machine. >>>>>>>> >>>>>>>> The second test case (ExceptionBarrierTest) is configured to use a >>>>>>>> custom ExceptionBarrier endpoint in place of DeadLetterChannel. Th= e >>>>>>>> ExceptionBarrierComponent is derived from a Camel DirectComponent >>>>>>>> with >>>>>>>> some custom code to handle Exchanges suffering from exceptions and >>>>>>>> forwarding the erroneous exchanges to the dead letter queue. With = the >>>>>>>> ExceptionBarrier, which takes in its simplistic implementation >>>>>>>> advantage >>>>>>>> of our specific routes to some extent, no messages are lost. >>>>>>>> >>>>>>>> The unit test are attached and I'd be most interested in learning >>>>>>>> what >>>>>>>> is going wrong. >>>>>>>> >>>>>>>> Cheers, Christian. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Claus Ibsen >>>>>>> Apache Camel Committer >>>>>>> >>>>>>> Open Source Integration: http://fusesource.com >>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238= p25326799.html >>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Claus Ibsen >>>>> Apache Camel Committer >>>>> >>>>> Open Source Integration: http://fusesource.com >>>>> Blog: http://davsclaus.blogspot.com/ >>>>> Twitter: http://twitter.com/davsclaus >>>>> >>>>> >>>> >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/Lost-Messages-in-DeadLetterChannel.-tp25289238p2= 5328848.html >>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>> >>>> >>> >>> >>> >>> -- >>> Claus Ibsen >>> Apache Camel Committer >>> >>> Open Source Integration: http://fusesource.com >>> Blog: http://davsclaus.blogspot.com/ >>> Twitter: http://twitter.com/davsclaus >>> >>> >> >> -- >> View this message in context: http://www.nabble.com/Lost-Messages-in-Dea= dLetterChannel.-tp25289238p25335523.html >> Sent from the Camel - Users mailing list archive at Nabble.com. >> >> > > > > -- > Claus Ibsen > Apache Camel Committer > > Open Source Integration: http://fusesource.com > Blog: http://davsclaus.blogspot.com/ > Twitter: http://twitter.com/davsclaus > --=20 Claus Ibsen Apache Camel Committer Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus