camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vitalii Tymchyshyn <>
Subject Enhancing non-blocking & reactiv capabilities of Camel
Date Wed, 27 Jul 2016 03:00:51 GMT

I work with Apache Camel for few years already and did some minor
enhancements in the past (e.g. enhancing AVRO support).
Now we have a new requirement for high-TPS fintech processing. Such TPS can
only being achieved with non-blocking processing. It turned out that camel
does a pretty decent job with it's async support. Yet there are certain
quirks and limitation and I'd like to spend some time improving camel
I already did a small straighforward PR (, but
before going forward, I'd like to discuss what and how to do.
Here are the directions I can see now:

1) Integration with Java 8 CompletableFuture. It's pretty straightforward,
I even created a and did a sending
You can do something like
CompletableFuture<Exchange> future = sender.sendBody("aaa");
For processing, I can easily do an abstract class that allows you to
protected abstract CompletableFuture<?> process(Exchange e) to make an
async processor
The problem is that I don't see how this can be integrated into camel
itself as it's on Java 7. What are the plans for Java 8?

2) ProducerTemplate now do not allow to perform a non-blocking call because
 a) Mandatory usage of ExecutorService
 b) Delegating any call to a synchronous send method:
public Future<Exchange> asyncSend(final Endpoint endpoint, final Processor
processor) {
    Callable<Exchange> task = new Callable<Exchange>() {
        public Exchange call() throws Exception {
            return send(endpoint, processor);
    return getExecutorService().submit(task);
The solution here can be to
a) Allow to easily use direct executor (executor that executes in the
original thread),
e.g. with setUseDirectExxecutor(true)
b) Implementing a major refactoring to perform such calls in non-blocking
async manner with callbacks.
Good thing would be to actually return a CompletableFuture in J8
Novadays the easiest way I found to perform a non-blocking call (I use it
in my J8 bridge) is to make a SendProcessor.

3) Bean language do not have support for callback-based processing beans.
I'd really like to see an @AsyncHandler and proper support for
AsyncProcessor in bean:ref URIs.
It would be great in J8 environment to allow @AsyncHandler method to return
a CompletableFuture instead of calling a handler.
It's still a question what to do for non-annotated cases. May be return
type of CompletableFuture could indicate it's async.

4) We are using Netty and currently Netty4 component does connection in a
blocking way. Among others,
it triggers a deadlock exception if you do two netty calls in one route. In
this case a connect can be called
from a netty callback thread. I am actually using a customm netty setup to
allow using usingExecutorService
from client side. I'd
a) Allow usingExecutorService from netty client. It's very useful if you do
any blocking activities in response processing
b) Make netty4 channel opening non-blocking. This would require to put
Future<Channel> instead of Channel into a generic pool.

Before I start, I will be very grateful for any comments / suggestions/
In general, I am happy to have ~10K tps of transactions going through camel
right now. One of the limitations is a string concatenation in a
instrumentation processor I am also looking forward to remove :)

Best regards, Vitalii Tymchyshyn

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message