camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hadrian Zbarcea <>
Subject Re: [DISCUSS - Camel 2.0 - Internal API reworkings] - Channel and AsyncProcessor
Date Fri, 24 Apr 2009 14:22:56 GMT
Agree, we should not remove anything until we have a solution.   
Personally I'd like to see the async part fixed in 2.0.
One of the reasons for M2 is also to give us a bit more time to get it  

+1 for Channel, if it matters :)


On Apr 24, 2009, at 8:38 AM, James Strachan wrote:

> I'm not yet convinced we should remove something, without another
> thing to put in its place. e.g. why don't we figure out a cleaner
> model and just switch to it (say at 2.1).
> For sure not that many things use async dispatch and it does cause
> confusion debugging; but there's definitely been folks with HTTP / WS
> / JBI async requirements over the last couple of years and
> AsyncProcessor is our only solution so far; so I'd rather leave it
> there - even if its sub optimal - until it can be replaced by a simple
> thing. Since its a purely internal API, I don't see that we have to
> definitely fix this for 2.0
> 2009/4/24 Claus Ibsen <>:
>> On Fri, Apr 24, 2009 at 10:17 AM, Willem Jiang < 
>> > wrote:
>>> +1 for introducing Channel into Camel world.
>>> For the AsyncProcessor part, it's is really difficult to understand.
>>> I think it is more than Call back, it will not block the calling  
>>> thread
>>> which will be used in the camel-jhc component[1].
>>> And I also did an enhancement[2] on the ErrorHandler of
>>> DeadLetterChannel by leveraging the AsyncProcessor to avoid the  
>>> blocking
>>> the calling thread when the DeadLetterChannel waits for another  
>>> retry.
>>> If the UnitOfWork can address these, I'm happy to give my +1 for
>>> removing the AsyncProcessor. If not, we need to find out a  
>>> replacement
>>> before removing it.
>>> [1]
>>> [2]
>> Yeah the AsyncProcessor and the callback notion currently implemented
>> in Camel is flawed, brittle, broken, not used at all from end user
>> perspective
>> and it started to spread out into the code base where it should not  
>> have.
>> I really think it should be removed, and if we need it, a new Async
>> API introduced that resemble the JDK async support (for instance the
>> Future).
>> To fix it I really think we need to
>> 1) Remove the existing AsyncProcessor
>> 2) At a later stage add a new API that leverages the JDK API.
>> About redelivery using different threads
>> =============================
>> Note the using another thread for redeliver is only feasable for non
>> transacted routes.
>> Transacted routes tend to depend on ThreadLocal and reuse session and
>> other stuff.
>> So the redeliver using different thread is not always desirable.
>> Note: But the redelivery is only supported by DeadLetterChannel that
>> does not support transacted routes anyway.
>> So you can say we are safe here.
>> But the end user does not have any choice to configure what they  
>> want.
>> I do think that end users should have the choice which model they
>> would like Camel to use for redelivery handling.
>> And with the Channel we would have another possiblity for redelivery
>> as we could just "route the message" back to the previous channel
>> and let it have some "delay time" before its "visible" on the channel
>> queue for re-processing.
>> So Willam, it would be much easier to impl. the use different thread
>> for redelivery when the Channel is more enhanced in Camel.
>> For instance for InOnly routes where there are no caller waiting we
>> can safely use others threads for redeliver (if not transacted)
>> But for InOut we need to add our own barrier that waits until the
>> UnitOfWork is complete before we can return the response.
>> Again I cannot stress too much that the camel-core API needs this
>> cleanup. You will get to this conclusion if you have been working 8h
>> around the clock in the camel-core code for as long as I have.
>>> Willem
>>> Claus Ibsen wrote:
>>>> Hi Camel riders
>>>> As you know we are working on Camel 2.0 and we decided to do a 2nd
>>>> milestone release.
>>>> This gives us more time to start on some of the ideas and internal
>>>> refactorings I wanted
>>>> to do for Camel 2.1.
>>>> After having dived really deed in the camel core codebase for the  
>>>> last
>>>> 6+ months or longer, I
>>>> feel that we need to cease the moment and do a more extensive house
>>>> cleaning before the Camel 2.0
>>>> release. The house cleaning is only internal and will not affect  
>>>> end
>>>> users of Camel.
>>>> Here are 2 issues I would like to address in the foreseeable  
>>>> future.
>>>> 1) Channel
>>>> ==========
>>>> To introduce a Channel between each node in the route path. The
>>>> channel is a composite processor
>>>> that is responsible for routing the exchange to the next node in  
>>>> the path.
>>>> We already do this but there is no visible notion of a Channel. The
>>>> "Channel" exists today as
>>>> a series of interceptors and an error handler. Today this is  
>>>> wrapped
>>>> at route build time
>>>> and then we have a static list of processors the exchanges is  
>>>> routed
>>>> through a runtime.
>>>> So what I am working on is to composite this series of interceptors
>>>> and error handler into a
>>>> DefaultChannel class that has the logic to do needed work before an
>>>> Exchange is routed to the next
>>>> node.
>>>> I have already discussed this a bit with Gert and James. James  
>>>> agreed
>>>> that it a was time for introducing
>>>> the Channel into Camel.
>>>> So I have started experimenting with this and got it working  
>>>> locally.
>>>> The Channel is in this first
>>>> stage just wrapping the existing series of interceptors and error
>>>> handler and thus the route logic
>>>> is not affect.
>>>> In the future the Channel will benefit in many ways as we can at
>>>> runtime add routing logic for instance
>>>> - enabling/disabling tracing (and other interceptors) at runtime
>>>> without reloading routes
>>>> - adding or removing interceptors at runtime without reloading  
>>>> routes
>>>> - adding or removing JMX performance metrics at runtime without  
>>>> reloading routes
>>>> - we could use it to blocking routing
>>>> - enhanced tooling support
>>>> - potential you could persist exchanges at the channel
>>>> - later add true async processing with the help of the channel
>>>> - easier to traverse the runtime route graph as we can traverse the
>>>> Channel, where we can have next/prev or the like methods
>>>> - and much more we can imagine
>>>> 2) AsyncProcessor
>>>> =================
>>>> I propose to remove the AsyncProcessor all together. I had a chat  
>>>> with
>>>> James about it and it was an experiment by Hiram back in early  
>>>> 2007.
>>>> Basically he had not worked on the code since, and its not really  
>>>> put
>>>> into good use. But sadly over time the AsyncProcessor have spread
>>>> itself into other core parts of Camel.
>>>> The basic idea was to attach a callback to a route so you could do
>>>> some "commit" work after the exchange is finished.
>>>> This idea is actually superseded by the UnitOfWork (introduced by
>>>> James) that is better for this kind of work. And we have schduled  
>>>> an
>>>> overhaul for UnitOfWork
>>>> in Camel 2.1 to allow more of our components to take advantage of  
>>>> this
>>>> and also expose DSLs for end users to attach their custom  
>>>> processors
>>>> or route.
>>>> The code in Camel core that is affected by the AsyncProcessor is  
>>>> much
>>>> more complex than regular Processor. In fact there are some areas
>>>> where its not
>>>> used correctly and causes unforseen side effects that only  
>>>> surfaces in
>>>> some complex or rare unit tests. This issues is more apparent  
>>>> lately
>>>> as more
>>>> and more camel processors supports the async processor directly.
>>>> The original code by Hiram only leverages the async callback in the
>>>> file component as it was part of his initial code. No other areas
>>>> benefits from this.
>>>> The code is basically making it bloat and complex inside Camel  
>>>> itself.
>>>> End users do not use this. I do not recall a single question on  
>>>> it in
>>>> the user forum.
>>>> So I am in a big +1 to get it out of the codebase.
>>>> And before you say well how do we do async routing then? Well the  
>>>> seda
>>>> component is still there to truely spawn a new thread to route an
>>>> exchange.
>>>> And we have the UnitOfWork where you are supposed to register your
>>>> callback. We might make the UnitOfWork a bit easier to register
>>>> callbacks for
>>>> only commit or failure depending on what you want. If its a single
>>>> methods its also easier for dynamic languages to have nice DSL  
>>>> support
>>>> for it.
>>>> Well that is just me rambling now..... And the UnitOfWork was James
>>>> envision for doing async work after the exchange is ended.
>>>> Summary
>>>> =======
>>>> 1) Channel
>>>> The first cut of this is already done on my local laptop and it  
>>>> will
>>>> just work as is today. But its bases the foundation for more exotic
>>>> and advanced
>>>> stuff in Camel 2.x series.
>>>> Already in progress.
>>>> Any thoughts?
>>>> 2) AsyncProcessor
>>>> To reduce the complexity for maintaining the Camel codebase. Just  
>>>> as
>>>> Gert if he was a bit puzzled how to do his StreamCache that uses  
>>>> the
>>>> Async stuff.
>>>> When we get it out of there the codebase is much easier to maintain
>>>> and we have a greater chance to attract potential new committers.
>>>> I vote +1 for this.
>>>> Any thoughts?
>> --
>> Claus Ibsen
>> Apache Camel Committer
>> Open Source Integration:
>> Blog:
>> Twitter:
>> Apache Camel Reference Card:
> -- 
> James
> -------
> Open Source Integration

View raw message