Mailing-List: contact dev-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@camel.apache.org
Message-ID: <254861263.1243436211209.JavaMail.jira@brutus>
Date: Wed, 27 May 2009 07:56:51 -0700 (PDT)
From: "Oliver Hecker (JIRA)" <jira@apache.org>
To: camel-dev@activemq.apache.org
Subject: [jira] Commented: (CAMEL-1650) Race condition in IdempotentConsumer
In-Reply-To: <1133870737.1243427990802.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/activemq/browse/CAMEL-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51944#action_51944 ] 

Oliver Hecker commented on CAMEL-1650:
--------------------------------------

Yes, I see the problem. The way how it is implemented at the moment assures an "at least once" quality of service.
Assuming that race conditions will not occur quite often this seems to be OK for most cases.
I was testing this because I am looking for a way to manage scheduled jobs (via Camel Quartz) in a clustered szenario
running the same camel config on multiple host in parallel. I am looking for a way to assure that even if the job is triggered
on every host only one of them gets actually processed. Due to the timers running more or less synchronous on all hosts
I am expecting that race conditions might occur quite often. (I actually also investigated in using the cluster features of Quartz
itself but found it not working within Camel Quartz)

Removing the id in case of a failure will not work if there there is some (technical) problem which prevents from deleting the entry
in the database. So if we crash after inserting in the database but before we could deliver to the destination then the message then
even when retrying later the filter will filter out the message.

I was thinking of some intermediate state to be stored in the repository to serve as a lock for the time between "contains" and "add".
But this requires quite some effort concerning coordination and detecting failed instances in the cluster.

So the Interface of IdempotentRepository might be something like

isProcessedAndLockIfNot (the former "contains"): 
  - Returns true if the id is contained in state "processed"
  - Returns false if id is not contained. Will insert id in state "locked"
  - if id is contained in state "locked" then the call will be blocked until
      - the id is either removed or goes to state processed, then proceed as above   or
      - a timeout is reached which indicates that the process which set the lock possibly crashed; in this case the lock will be stolen and false will be returned

setProcessed (the former "add"):
  - set state of ID to "processed"

unlock (new):
  - delete id from the repository;

It might even be necessary to keep track of who aquired a lock to avoid messing up if the locker tries to release a lock after the timeout occurred.

In any case this requires quite some interface and coding changes (and some more thoughts about how to do it).


> Race condition in IdempotentConsumer
> ------------------------------------
>
>                 Key: CAMEL-1650
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1650
>             Project: Apache Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 2.0-M1
>            Reporter: Oliver Hecker
>         Attachments: IdempotentConsumerTest.java
>
>
> A possible possible race condition exists in the IdempotentConsumer implementation:
> The code first checks in the MessageIdRepository if the message was already processed. If not then it processes the message and
> afterwards adds the id to the repository. (See also http://issues.apache.org/activemq/browse/CAMEL-1451). There is no locking
> between the check with "contains" and the insert with "add". So if multiple threads/instances try this in parallel for the same id, then
> it might happen that more than one finds the id not yet contained in the repository and the same message is processed multiple
> times.
> I enclose an extended version of IdempotentConsumerTest which illustrates the problem.
> It is important to note that even if the test demonstrates the issue with an MemoryIdempotentRepository a solution should also
> address the case of a database based respository in a clustered environment. So this might imply that some locking mechanism on the
> database is required.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.