camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raul Kripalani <>
Subject Re: [DISCUSS] How to better enlist the duplicate message count by the IdempotentConsumer on JMX
Date Thu, 05 Jan 2012 17:58:12 GMT
Responding to Claus' request for feedback on how to store EIP stats and
expose it in several ways...

With the current architecture where stats are kept in memory there are a
few downsides, mainly: (1) not being able to store rich data, only counters
due to memory usage considerations and (2) stats are lost between restarts.

A legitimate requirement is to be able to browse the list of exchange IDs
marked as duplicates by the idempotent consumer, but storing those in
memory would cause an immediate memory leak due to an ever-growing list.
Good thing is that at least we can set skipDuplicate to false and react to
the DUPLICATE_MESSAGE storing the information somewhere, but it would be
easier and less error prone if there was a centralised place to resort to
to obtain better richer visibility and governance, without the user having
to implement hooks here and there to react when things happen inside Camel
(tracer, interceptor, etc.).

In my mind, the ideal solution is to create a generic "management engine"
to which components, EIPs, services, etc. publish their stats. This
represents an important overhaul, but in my opinion it is imperative for
Camel to advance to the next generation ;)

Features of my proposed "management engine":

* pluggable persistence repositories, to choose what back-end your stats
will be persisted to (JDBC, MongoDB, Redis, In-memory, etc.)
* flexible external access via pluggable management front-ends (JMX, REST,
raw API), so that any type of client can access the info from the outside
* persistence flushing policy, so that the user can control how often the
management engine will persist in-memory data to the back-end
* purge policy, so that old stats are flushed out on a schedule, i.e. it's
* centralised filtering of management events, so you can turn off stats for
specific EIPs, Components, etc. from one point only. E.g. (*,
!idempotentConsumer, !camel-jpa) would enable all stats except idempotent
consumer and the camel-jpa component.
* query capabilities based on the Simple language
* async/sync persistence behaviour selection ... based on how important
your stats are to you
* etc.

Note that achieving blazing-fast persistence nowadays is a piece of cake
with NoSQL, Big Data and Data Grids.

I realise the extent of the impact on the codebase to implement this
feature, but seeing that Camel 3.0 is coming up in conversations, I wanted
to drop my 2 cents ;)


On 5 January 2012 16:14, Babak Vahdat <> wrote:

> As Claus has already suggested [1] I would like to share the question with
> you for a better possible third approach!
> For all the details regarding the proposed improvement please take a look
> at
> the comments and also the 2 attached patches on it and let us know of any
> possible third approach you could think of. Thanks!
> [1]
> Babak
> --
> View this message in context:
> Sent from the Camel Development mailing list archive at

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message