activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Hubert (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-3597) Enable RedeliveryPolicy to determine next delay independent of the consumer
Date Sat, 19 Nov 2011 13:59:52 GMT

    [ https://issues.apache.org/jira/browse/AMQ-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153495#comment-13153495
] 

Eric Hubert commented on AMQ-3597:
----------------------------------

Timothy, you are absolutely right that we did not associate the term redelivery with the meaning
consumer internal reprocessing (using a consumer-side queue) and redispatch with broker to
consumer and thus may have used them interchangeable potentially leading to confusion.

Stepping back explaining the use case and problem to solve might indeed be the best proposal
I'd like to follow hereby.

1) Use Case/Problem Context:
In our use case the consumer has the responsibility to send the message polled from the broker's
queue to http web service endpoints and ensure the message has reached the endpoint it was
meant for. This receiving endpoint may not always be available. So a configurable number of
retries with a configurable non-constant, increasing delay shall take place before considering
the message as non-deliverable at some point (DLQ use). Within this time even the rules determining
the actual service endpoint may change.
I think at this point I need to add that this whole process is taking place as mediation logic
inside an ESB product integrating any JMS message broker via Spring. Sending to the endpoint
is done using non-blocking http transports. 
The setup must be highly available and process high message volumes. Messages must never get
lost.  The ESB nodes are the only consumers on those JMS queues, so it is a rather very small
number of consumers.
So consequently there are multiple ESB nodes and multiple brokers in master slave setup. Ideally
the delays between send attempts to the service endpoints should follow the calculation independent
of an ESB node (JMS consumer) unavailability (due to maintenance, crash or whatever reason)
or broker unavailability (master slave failover should take place transparently). 
So from a business perspective we only care for the full transfer between JMS broker and service
endpoint and consider this as the delivery process in this context. The same applies for redelivery
attempts.

2) Current solution approach:
- The ESB basically uses Spring’s DefaultMessageListenerContainer (DMLC) in conjunction
with Spring’s PlatformTransactionManager  (PTM).
-The message received from the queue arrives at the incoming message sequence of the mediation
logic starting a transaction. If there are no "routing rules" an immediate rollback will be
performed to trigger a scheduled redelivery. Whether this is done from the same consumer (ESB
node) or any should not play a role. In case routing rules are available the transaction will
be suspended and the message will be send to the service endpoint. ESB-internally this is
an asynchronous, non-blocking operation. At some point either an error occurs or a response
message arrives which will be inspected in the out message sequence in order to resume the
transaction and either perform a commit (on success) or a rollback (on failure) to schedule
a resend (intentionally trying to avoid the terms redelivery or redispatch here).

3) Issues with current solution approach:
If we got it right AMQ redelivery feature is consumer-based requiring a transacted session
AND the same consumer instance to be used for redeliveries. In terms of concrete DMLC configuration
this  requires cache level CACHE_CONSUMER as well as sessionTransacted set to true.
This seems to make it impossible to have control over the transaction and use the PTM to suspend,
resume, commit or rollback a message.
Using only CACHE_CONSUMER with PTM does not work (session not transacted, no reaction on rollback).
Using any other cache level, the transaction handling is working fine, the messages are also
scheduled for redelivery, but the message to be redelivered is redispatched from the broker
(likely a result of a consumer close in DMLC) resulting in another undelayed delivery attempt.
So infact we do not get our desired transactional behavior plus redelivery to work. In addition,
even if we could get this to work properly I assume a consumer crash or shutdown while a redelivery
process is going on (which could be configured to take place over multiple days with maximum
delays of an hour) would result in an automatic rollback and a message redispatch to another
consumer starting from scratch with an immediate (undelayed) delivery irrespective of the
redelivery counter on the message and the delay which shall take place here. 
If understanding AMQ-2710 correctly and another non-configurable behavior of consumer based-redelivery
with strict JMS semantics is always a blocking of consumption of further messages  in case
of a running redelivery process (to guarantee the order). This would be another reason why
we could not use this approach unless we would use a separate queue for each individual logical
routing target as any problem sending to one service endpoint must not block message sending
to other endpoints.
So indeed it looks like AMQ-2710 could be an alternative as this seems to be a realization
of broker -based delayed redelivery. I know, for the sake of clarity one should rather call
a solution with broker -based delayed redispatch loosing order guarantee. 
I will further investigate this suggestion (also checking the linked old JIRA issue AMQ-1853),
but still I don’t get why this simple use case seems to be so hard to realize and why I
have to integrate an EIP framework like Camel for this to work around.

If there is some way to get DMLC with CACHE_CONSUMER, local transactions via PTM and redelivery
to work properly, which we tried hard but failed, this would of course also still be interesting.

Timothy, in any case many thanks for this feedback! It was the first one moving us slightly
forward in terms of understanding of the AMQ implementation.
                
> Enable RedeliveryPolicy to determine next delay independent of the consumer
> ---------------------------------------------------------------------------
>
>                 Key: AMQ-3597
>                 URL: https://issues.apache.org/jira/browse/AMQ-3597
>             Project: ActiveMQ
>          Issue Type: Improvement
>          Components: Broker
>    Affects Versions: 5.5.1
>            Reporter: Eric Hubert
>            Assignee: Timothy Bish
>             Fix For: 5.6.0
>
>
> Currently the redelivery is bound to the consumer instance requiring the end user to
use the same message consumer instance. This is due to the fact that the redelivery delay
is computated based on the previous redelivery delay (stored with the consumer instance) and
not based on the redeliveryCount of the message.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message