activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adrian Trenaman (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQ-2803) 'Zombie' messages created in KahaDB after failover, with warning 'Duplicate message add attempt rejected.'
Date Mon, 28 Jun 2010 11:31:52 GMT

    [ https://issues.apache.org/activemq/browse/AMQ-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=60308#action_60308
] 

Adrian Trenaman commented on AMQ-2803:
--------------------------------------

As an alternative, I've tried the same test using the {{<amqPersistenceAdapter>}} instead,
with the configuration below. 

{code:xml}
		<persistenceAdapter>
			<!-- <kahaDB directory="${activemq-broker.dataDirectory}" /> -->
			<amqPersistenceAdapter directory="${activemq-broker.dataDirectory}" syncOnWrite="true"
maxFileLength="32mb"/>
		</persistenceAdapter>
{code}

Using this adapter instead of KahaDB, I find that I don't get the warning message. However,
I *do* get the same 'zombie' messages in JMX: the queueSize suggests that there are messages
to be delivered (for example, in two failover tests I ended up with a JMX {{QueueSize}} of
'3' and '1' respectively after failover), when in fact all messages were delivered successfully.
Nicely, the {{<amqPersistenceAdapter>}} is better behaved in that if I restart the broker
then the {{QueueSize}} is reported correctly again. 

> 'Zombie' messages created in KahaDB after failover, with warning 'Duplicate message add
attempt rejected.'
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: AMQ-2803
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2803
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.3.2
>            Reporter: Adrian Trenaman
>
> Am doing some failover testing of ActiveMQ, sending 10,000 messages to the broker with
a live consumer; the producer and consumer are transactional, and the broker is configured
to use KahaDB. On failover of the broker, I see log warnings like this: 
> {code}
>  WARN | Duplicate message add attempt rejected. Message id: ID:Ade's-54024-1277715585703-0:29:1:1:240
>  WARN | Duplicate message add attempt rejected. Message id: ID:Ade's-54024-1277715585703-0:24:1:1:241
>  WARN | Duplicate message add attempt rejected. Message id: ID:Ade's-54024-1277715585703-0:23:1:1:242
>  WARN | Duplicate message add attempt rejected. Message id: ID:Ade's-54024-1277715585703-0:0:1:1:237
> {code}
> When the test has run its course, I discover that there are no messages lost - all my
messages have gone through successfully. However, I notice that there are in fact a number
of 'zombie' messages on the queue: by 'zombie' I mean that the messages are in the store,
and the JMX queueSize says they're there. However, if you browse the queue you don't see them,
and, if you have a consumer on the queue, you won't receive them either. 
> Looking into the code, in {{org.apache.activemq.store.kahadb.MessageDatabase}} you can
see the following very interesting comments concerning the log warning:
> {code}
>                // If the message ID as indexed, then the broker asked us to store
a DUP
>                 // message.  Bad BOY!  Don't do it, and log a warning.
>                 LOG.warn("Duplicate message add attempt rejected. Message id:
"+command.getMessageId());
>                 // TODO: consider just rolling back the tx.
>                 sd.messageIdIndex.put(tx, command.getMessageId(), previous);
> {code} 
> So. It seems to me that while we're detecting the condition of the broker trying to reinsert
a message that's already in the KahaDB correctly, we're not handling this case very well.
The outcome is that while we do not get any message loss, we do get worrying warning messages
*and* we get erroneous reporting of the queueSize in JMX which will mislead and create the
perception of undelivered messages.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message