camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bengt Rodehav <be...@rodehav.com>
Subject Re: [DISCUSS] How to better enlist the duplicate message count by the IdempotentConsumer on JMX
Date Fri, 06 Jan 2012 10:22:49 GMT
This sound really interesting.

Currently our legacy products are integrated with external applications
using an integration platform that has been developed in-house. We are now
creating an integration platform that is Camel based instead. One of the
things we had support for in our old integration platform was a message
repository. All messages (incoming and outgoing) were stored in a database
along with information showing whether the message was processed
successfully or not. We also have a rich user interface for finding and
inspecting messages as well as resending messages that have failed.

I think this is required functionality for any integration platform. So
far, this has not been part of Camel and I have envisioned to keep
developing this functionality in house. It seems like you are approaching
this from the statistics side (which is fine) but if you broaden the
concept to a complete message repository along with querying and resending
functionality then this could be extremely useful.

/Bengt

2012/1/6 Charles Moulliard <cmoulliard@gmail.com>

> Hi Raul,
>
> This is a brillant feature that you propose here and I think that the time
> is perfect (with up coming Camel 3.0 release)
> that we discuss as we need such stats in projects. I think that the plugin
> strategy proposed is fine and can be included
> in the camel-core module for in-memory usage.
> Additional implementations requiring DB, NoSQL storage should be part of a
> new camel module (maybe) called camel-extra or camel-extend containing
> different plugins one by implementation (JDBC, JPA, Hazelcast, MongoDb,
> Cassandra, ...).
> Some questions come to me like :
> 1) What is the definition of the "stats" or more precisely which info will
> be part of the stats (camel JMX stats, camel exchangeId, camel bredCrumbId,
> id collected by Idempotent - Aggregator - Splitter processors, trace in
> text, XML formats ...) ?
> 2) Do we need one plugin by storage engine or one plugin by strategy (=
> what we would like to store : trace, processors exchange, ....) and storage
> engine ?
> 3) Which kind of granularity do we want - all, processors, (= idempotent,
> splitter, ...), idempotent, splitter, trace (aka text or XML messages that
> usually we send to logger, ...) ?
> 4) Which solution are we gonna to use to add this plugin strategy (event
> notifier + plugin, ....)
>
> Remark : It could be interesting that you create a JIRA ticket to continue
> this discussion ;-)
>
> Regards,
>
> Charles
>
> people in charge of the operations and business users
> On Thu, Jan 5, 2012 at 6:58 PM, Raul Kripalani <raul@fusesource.com>
> wrote:
>
> > Responding to Claus' request for feedback on how to store EIP stats and
> > expose it in several ways...
> >
> > With the current architecture where stats are kept in memory there are a
> > few downsides, mainly: (1) not being able to store rich data, only
> counters
> > due to memory usage considerations and (2) stats are lost between
> restarts.
> >
> > A legitimate requirement is to be able to browse the list of exchange IDs
> > marked as duplicates by the idempotent consumer, but storing those in
> > memory would cause an immediate memory leak due to an ever-growing list.
> > Good thing is that at least we can set skipDuplicate to false and react
> to
> > the DUPLICATE_MESSAGE storing the information somewhere, but it would be
> > easier and less error prone if there was a centralised place to resort to
> > to obtain better richer visibility and governance, without the user
> having
> > to implement hooks here and there to react when things happen inside
> Camel
> > (tracer, interceptor, etc.).
> >
> > In my mind, the ideal solution is to create a generic "management engine"
> > to which components, EIPs, services, etc. publish their stats. This
> > represents an important overhaul, but in my opinion it is imperative for
> > Camel to advance to the next generation ;)
> >
> > Features of my proposed "management engine":
> >
> > * pluggable persistence repositories, to choose what back-end your stats
> > will be persisted to (JDBC, MongoDB, Redis, In-memory, etc.)
> > * flexible external access via pluggable management front-ends (JMX,
> REST,
> > raw API), so that any type of client can access the info from the outside
> > * persistence flushing policy, so that the user can control how often the
> > management engine will persist in-memory data to the back-end
> > * purge policy, so that old stats are flushed out on a schedule, i.e.
> it's
> > self-cleaning
> > * centralised filtering of management events, so you can turn off stats
> for
> > specific EIPs, Components, etc. from one point only. E.g. (*,
> > !idempotentConsumer, !camel-jpa) would enable all stats except idempotent
> > consumer and the camel-jpa component.
> > * query capabilities based on the Simple language
> > * async/sync persistence behaviour selection ... based on how important
> > your stats are to you
> > * etc.
> >
> >
> > Note that achieving blazing-fast persistence nowadays is a piece of cake
> > with NoSQL, Big Data and Data Grids.
> >
> > I realise the extent of the impact on the codebase to implement this
> > feature, but seeing that Camel 3.0 is coming up in conversations, I
> wanted
> > to drop my 2 cents ;)
> >
> > Regards,
> > Raúl.
> >
> > On 5 January 2012 16:14, Babak Vahdat <babak.vahdat@swissonline.ch>
> wrote:
> >
> > > As Claus has already suggested [1] I would like to share the question
> > with
> > > you for a better possible third approach!
> > >
> > > For all the details regarding the proposed improvement please take a
> look
> > > at
> > > the comments and also the 2 attached patches on it and let us know of
> any
> > > possible third approach you could think of. Thanks!
> > >
> > > [1] https://issues.apache.org/jira/browse/CAMEL-4782
> > >
> > > Babak
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://camel.465427.n5.nabble.com/DISCUSS-How-to-better-enlist-the-duplicate-message-count-by-the-IdempotentConsumer-on-JMX-tp5123118p5123118.html
> > > Sent from the Camel Development mailing list archive at Nabble.com.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message