camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Strachan <>
Subject Re: [DISCUSS - Camel 2.0 - Internal API reworkings] - Channel and AsyncProcessor
Date Fri, 24 Apr 2009 12:38:25 GMT
I'm not yet convinced we should remove something, without another
thing to put in its place. e.g. why don't we figure out a cleaner
model and just switch to it (say at 2.1).

For sure not that many things use async dispatch and it does cause
confusion debugging; but there's definitely been folks with HTTP / WS
/ JBI async requirements over the last couple of years and
AsyncProcessor is our only solution so far; so I'd rather leave it
there - even if its sub optimal - until it can be replaced by a simple
thing. Since its a purely internal API, I don't see that we have to
definitely fix this for 2.0

2009/4/24 Claus Ibsen <>:
> On Fri, Apr 24, 2009 at 10:17 AM, Willem Jiang <> wrote:
>> +1 for introducing Channel into Camel world.
>> For the AsyncProcessor part, it's is really difficult to understand.
>> I think it is more than Call back, it will not block the calling thread
>> which will be used in the camel-jhc component[1].
>> And I also did an enhancement[2] on the ErrorHandler of
>> DeadLetterChannel by leveraging the AsyncProcessor to avoid the blocking
>> the calling thread when the DeadLetterChannel waits for another retry.
>> If the UnitOfWork can address these, I'm happy to give my +1 for
>> removing the AsyncProcessor. If not, we need to find out a replacement
>> before removing it.
>> [1]
>> [2]
> Yeah the AsyncProcessor and the callback notion currently implemented
> in Camel is flawed, brittle, broken, not used at all from end user
> perspective
> and it started to spread out into the code base where it should not have.
> I really think it should be removed, and if we need it, a new Async
> API introduced that resemble the JDK async support (for instance the
> Future).
> To fix it I really think we need to
> 1) Remove the existing AsyncProcessor
> 2) At a later stage add a new API that leverages the JDK API.
> About redelivery using different threads
> =============================
> Note the using another thread for redeliver is only feasable for non
> transacted routes.
> Transacted routes tend to depend on ThreadLocal and reuse session and
> other stuff.
> So the redeliver using different thread is not always desirable.
> Note: But the redelivery is only supported by DeadLetterChannel that
> does not support transacted routes anyway.
> So you can say we are safe here.
> But the end user does not have any choice to configure what they want.
> I do think that end users should have the choice which model they
> would like Camel to use for redelivery handling.
> And with the Channel we would have another possiblity for redelivery
> as we could just "route the message" back to the previous channel
> and let it have some "delay time" before its "visible" on the channel
> queue for re-processing.
> So Willam, it would be much easier to impl. the use different thread
> for redelivery when the Channel is more enhanced in Camel.
> For instance for InOnly routes where there are no caller waiting we
> can safely use others threads for redeliver (if not transacted)
> But for InOut we need to add our own barrier that waits until the
> UnitOfWork is complete before we can return the response.
> Again I cannot stress too much that the camel-core API needs this
> cleanup. You will get to this conclusion if you have been working 8h
> around the clock in the camel-core code for as long as I have.
>> Willem
>> Claus Ibsen wrote:
>>> Hi Camel riders
>>> As you know we are working on Camel 2.0 and we decided to do a 2nd
>>> milestone release.
>>> This gives us more time to start on some of the ideas and internal
>>> refactorings I wanted
>>> to do for Camel 2.1.
>>> After having dived really deed in the camel core codebase for the last
>>> 6+ months or longer, I
>>> feel that we need to cease the moment and do a more extensive house
>>> cleaning before the Camel 2.0
>>> release. The house cleaning is only internal and will not affect end
>>> users of Camel.
>>> Here are 2 issues I would like to address in the foreseeable future.
>>> 1) Channel
>>> ==========
>>> To introduce a Channel between each node in the route path. The
>>> channel is a composite processor
>>> that is responsible for routing the exchange to the next node in the path.
>>> We already do this but there is no visible notion of a Channel. The
>>> "Channel" exists today as
>>> a series of interceptors and an error handler. Today this is wrapped
>>> at route build time
>>> and then we have a static list of processors the exchanges is routed
>>> through a runtime.
>>> So what I am working on is to composite this series of interceptors
>>> and error handler into a
>>> DefaultChannel class that has the logic to do needed work before an
>>> Exchange is routed to the next
>>> node.
>>> I have already discussed this a bit with Gert and James. James agreed
>>> that it a was time for introducing
>>> the Channel into Camel.
>>> So I have started experimenting with this and got it working locally.
>>> The Channel is in this first
>>> stage just wrapping the existing series of interceptors and error
>>> handler and thus the route logic
>>> is not affect.
>>> In the future the Channel will benefit in many ways as we can at
>>> runtime add routing logic for instance
>>> - enabling/disabling tracing (and other interceptors) at runtime
>>> without reloading routes
>>> - adding or removing interceptors at runtime without reloading routes
>>> - adding or removing JMX performance metrics at runtime without reloading routes
>>> - we could use it to blocking routing
>>> - enhanced tooling support
>>> - potential you could persist exchanges at the channel
>>> - later add true async processing with the help of the channel
>>> - easier to traverse the runtime route graph as we can traverse the
>>> Channel, where we can have next/prev or the like methods
>>> - and much more we can imagine
>>> 2) AsyncProcessor
>>> =================
>>> I propose to remove the AsyncProcessor all together. I had a chat with
>>> James about it and it was an experiment by Hiram back in early 2007.
>>> Basically he had not worked on the code since, and its not really put
>>> into good use. But sadly over time the AsyncProcessor have spread
>>> itself into other core parts of Camel.
>>> The basic idea was to attach a callback to a route so you could do
>>> some "commit" work after the exchange is finished.
>>> This idea is actually superseded by the UnitOfWork (introduced by
>>> James) that is better for this kind of work. And we have schduled an
>>> overhaul for UnitOfWork
>>> in Camel 2.1 to allow more of our components to take advantage of this
>>> and also expose DSLs for end users to attach their custom processors
>>> or route.
>>> The code in Camel core that is affected by the AsyncProcessor is much
>>> more complex than regular Processor. In fact there are some areas
>>> where its not
>>> used correctly and causes unforseen side effects that only surfaces in
>>> some complex or rare unit tests. This issues is more apparent lately
>>> as more
>>> and more camel processors supports the async processor directly.
>>> The original code by Hiram only leverages the async callback in the
>>> file component as it was part of his initial code. No other areas
>>> benefits from this.
>>> The code is basically making it bloat and complex inside Camel itself.
>>> End users do not use this. I do not recall a single question on it in
>>> the user forum.
>>> So I am in a big +1 to get it out of the codebase.
>>> And before you say well how do we do async routing then? Well the seda
>>> component is still there to truely spawn a new thread to route an
>>> exchange.
>>> And we have the UnitOfWork where you are supposed to register your
>>> callback. We might make the UnitOfWork a bit easier to register
>>> callbacks for
>>> only commit or failure depending on what you want. If its a single
>>> methods its also easier for dynamic languages to have nice DSL support
>>> for it.
>>> Well that is just me rambling now..... And the UnitOfWork was James
>>> envision for doing async work after the exchange is ended.
>>> Summary
>>> =======
>>> 1) Channel
>>> The first cut of this is already done on my local laptop and it will
>>> just work as is today. But its bases the foundation for more exotic
>>> and advanced
>>> stuff in Camel 2.x series.
>>> Already in progress.
>>> Any thoughts?
>>> 2) AsyncProcessor
>>> To reduce the complexity for maintaining the Camel codebase. Just as
>>> Gert if he was a bit puzzled how to do his StreamCache that uses the
>>> Async stuff.
>>> When we get it out of there the codebase is much easier to maintain
>>> and we have a greater chance to attract potential new committers.
>>> I vote +1 for this.
>>> Any thoughts?
> --
> Claus Ibsen
> Apache Camel Committer
> Open Source Integration:
> Blog:
> Twitter:
> Apache Camel Reference Card:


Open Source Integration

View raw message