camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pmp.martins" <pmp.mart...@campus.fct.unl.pt>
Subject How to improve efficiency with Camel?
Date Sun, 23 Feb 2014 22:05:12 GMT
I have a problem of efficiency in my project which uses Camel with the  Esper
component <http://camel.apache.org/esper.html>  . 

I have several external datasources feeding information to camel endpoints.
Each Camel endpoint that receives data transfers it to a route that
processes it and then delivers it at an Esper endpoint.

The image below illustrates this behavior:

<http://camel.465427.n5.nabble.com/file/n5747836/cPVWQ.png> 

The efficiency problem is that all of this is done by a single Java thread.
Thus if I have many sources, there is a huge bottleneck. 


The following code accurately illustrates what is going on with the image:

    


*To fix this problem, I have to handle this situation with a pool of threads
or using some sort of parallel processing.* I cannot use patterns like
multicast, recipient list, etc because all of those send the same message to
multiple endpoints / clients, which is not the case in my examples.

A possible solution would be having 1 thread per each "Datasource endpoint
-> Route -> Esper endpoint" combination, like the image bellow:

<http://camel.465427.n5.nabble.com/file/n5747836/hHg6A.png> 

Another possible solution is to have 1 thread receive everything from the
datasources, and then dispatch it to multiple threads handling the route
processing together with the other endpoint:
<http://camel.465427.n5.nabble.com/file/n5747836/L2FgW.png> 

*PS:* I am open to any other possible suggestions you may have.

To achieve one of these I have considered using the Camel  SEDA component
<http://camel.apache.org/seda.html>   component, however, this one does not
seem to allow me to have dynamic thread pools, because the
`concurrentConsumers` property is static. 

Furthermore, I was alerted in StackOverflow for loss of messages using SEDA,
so I am not sure how trustworthy this component really is. Can someone
confirm?

So, at this point I am quite lost and I don't know what to do:
- Is SEDA the solution I am looking for?
- If yes, how do I integrate it with the Esper endpoint given the syntax
problem?
- Are there any other solutions / Camel components that could fix my
problem?




--
View this message in context: http://camel.465427.n5.nabble.com/How-to-improve-efficiency-with-Camel-tp5747836.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Mime
View raw message