camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Moulliard <>
Subject Re: [DISCUSS] How to better enlist the duplicate message count by the IdempotentConsumer on JMX
Date Fri, 06 Jan 2012 08:00:46 GMT
Hi Raul,

This is a brillant feature that you propose here and I think that the time
is perfect (with up coming Camel 3.0 release)
that we discuss as we need such stats in projects. I think that the plugin
strategy proposed is fine and can be included
in the camel-core module for in-memory usage.
Additional implementations requiring DB, NoSQL storage should be part of a
new camel module (maybe) called camel-extra or camel-extend containing
different plugins one by implementation (JDBC, JPA, Hazelcast, MongoDb,
Cassandra, ...).
Some questions come to me like :
1) What is the definition of the "stats" or more precisely which info will
be part of the stats (camel JMX stats, camel exchangeId, camel bredCrumbId,
id collected by Idempotent - Aggregator - Splitter processors, trace in
text, XML formats ...) ?
2) Do we need one plugin by storage engine or one plugin by strategy (=
what we would like to store : trace, processors exchange, ....) and storage
engine ?
3) Which kind of granularity do we want - all, processors, (= idempotent,
splitter, ...), idempotent, splitter, trace (aka text or XML messages that
usually we send to logger, ...) ?
4) Which solution are we gonna to use to add this plugin strategy (event
notifier + plugin, ....)

Remark : It could be interesting that you create a JIRA ticket to continue
this discussion ;-)



people in charge of the operations and business users
On Thu, Jan 5, 2012 at 6:58 PM, Raul Kripalani <> wrote:

> Responding to Claus' request for feedback on how to store EIP stats and
> expose it in several ways...
> With the current architecture where stats are kept in memory there are a
> few downsides, mainly: (1) not being able to store rich data, only counters
> due to memory usage considerations and (2) stats are lost between restarts.
> A legitimate requirement is to be able to browse the list of exchange IDs
> marked as duplicates by the idempotent consumer, but storing those in
> memory would cause an immediate memory leak due to an ever-growing list.
> Good thing is that at least we can set skipDuplicate to false and react to
> the DUPLICATE_MESSAGE storing the information somewhere, but it would be
> easier and less error prone if there was a centralised place to resort to
> to obtain better richer visibility and governance, without the user having
> to implement hooks here and there to react when things happen inside Camel
> (tracer, interceptor, etc.).
> In my mind, the ideal solution is to create a generic "management engine"
> to which components, EIPs, services, etc. publish their stats. This
> represents an important overhaul, but in my opinion it is imperative for
> Camel to advance to the next generation ;)
> Features of my proposed "management engine":
> * pluggable persistence repositories, to choose what back-end your stats
> will be persisted to (JDBC, MongoDB, Redis, In-memory, etc.)
> * flexible external access via pluggable management front-ends (JMX, REST,
> raw API), so that any type of client can access the info from the outside
> * persistence flushing policy, so that the user can control how often the
> management engine will persist in-memory data to the back-end
> * purge policy, so that old stats are flushed out on a schedule, i.e. it's
> self-cleaning
> * centralised filtering of management events, so you can turn off stats for
> specific EIPs, Components, etc. from one point only. E.g. (*,
> !idempotentConsumer, !camel-jpa) would enable all stats except idempotent
> consumer and the camel-jpa component.
> * query capabilities based on the Simple language
> * async/sync persistence behaviour selection ... based on how important
> your stats are to you
> * etc.
> Note that achieving blazing-fast persistence nowadays is a piece of cake
> with NoSQL, Big Data and Data Grids.
> I realise the extent of the impact on the codebase to implement this
> feature, but seeing that Camel 3.0 is coming up in conversations, I wanted
> to drop my 2 cents ;)
> Regards,
> Raúl.
> On 5 January 2012 16:14, Babak Vahdat <> wrote:
> > As Claus has already suggested [1] I would like to share the question
> with
> > you for a better possible third approach!
> >
> > For all the details regarding the proposed improvement please take a look
> > at
> > the comments and also the 2 attached patches on it and let us know of any
> > possible third approach you could think of. Thanks!
> >
> > [1]
> >
> > Babak
> >
> >
> > --
> > View this message in context:
> >
> > Sent from the Camel Development mailing list archive at
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message