camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Oliver Hecker (JIRA)" <>
Subject [jira] Commented: (CAMEL-1650) Race condition in IdempotentConsumer
Date Wed, 27 May 2009 14:56:51 GMT


Oliver Hecker commented on CAMEL-1650:

Yes, I see the problem. The way how it is implemented at the moment assures an "at least once"
quality of service.
Assuming that race conditions will not occur quite often this seems to be OK for most cases.
I was testing this because I am looking for a way to manage scheduled jobs (via Camel Quartz)
in a clustered szenario
running the same camel config on multiple host in parallel. I am looking for a way to assure
that even if the job is triggered
on every host only one of them gets actually processed. Due to the timers running more or
less synchronous on all hosts
I am expecting that race conditions might occur quite often. (I actually also investigated
in using the cluster features of Quartz
itself but found it not working within Camel Quartz)

Removing the id in case of a failure will not work if there there is some (technical) problem
which prevents from deleting the entry
in the database. So if we crash after inserting in the database but before we could deliver
to the destination then the message then
even when retrying later the filter will filter out the message.

I was thinking of some intermediate state to be stored in the repository to serve as a lock
for the time between "contains" and "add".
But this requires quite some effort concerning coordination and detecting failed instances
in the cluster.

So the Interface of IdempotentRepository might be something like

isProcessedAndLockIfNot (the former "contains"): 
  - Returns true if the id is contained in state "processed"
  - Returns false if id is not contained. Will insert id in state "locked"
  - if id is contained in state "locked" then the call will be blocked until
      - the id is either removed or goes to state processed, then proceed as above   or
      - a timeout is reached which indicates that the process which set the lock possibly
crashed; in this case the lock will be stolen and false will be returned

setProcessed (the former "add"):
  - set state of ID to "processed"

unlock (new):
  - delete id from the repository;

It might even be necessary to keep track of who aquired a lock to avoid messing up if the
locker tries to release a lock after the timeout occurred.

In any case this requires quite some interface and coding changes (and some more thoughts
about how to do it).

> Race condition in IdempotentConsumer
> ------------------------------------
>                 Key: CAMEL-1650
>                 URL:
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 2.0-M1
>            Reporter: Oliver Hecker
>         Attachments:
> A possible possible race condition exists in the IdempotentConsumer implementation:
> The code first checks in the MessageIdRepository if the message was already processed.
If not then it processes the message and
> afterwards adds the id to the repository. (See also
There is no locking
> between the check with "contains" and the insert with "add". So if multiple threads/instances
try this in parallel for the same id, then
> it might happen that more than one finds the id not yet contained in the repository and
the same message is processed multiple
> times.
> I enclose an extended version of IdempotentConsumerTest which illustrates the problem.
> It is important to note that even if the test demonstrates the issue with an MemoryIdempotentRepository
a solution should also
> address the case of a database based respository in a clustered environment. So this
might imply that some locking mechanism on the
> database is required.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message