activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Chernyakhovsky (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-5249) "cursor got duplicate" error after upgrade
Date Wed, 24 Dec 2014 10:20:14 GMT

    [ https://issues.apache.org/jira/browse/AMQ-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258159#comment-14258159
] 

Alex Chernyakhovsky commented on AMQ-5249:
------------------------------------------

Gary,

I will check the unit test module but I am afraid it might prove hard to replicate our environment.
Maybe it will be more productive if I work with you or someone assigned to this issue to understand
what exactly is happening and then proceed with the test.

Here is some info:
1) Our production system consists of about half a dozen of servers running clients producing
messages, then another set of about 10 consuming the load while producing some messages for
consumption by themselves and another set of about 5 servers running apache storm. (The numbers
change as we scale). 8 queues are created across the board.
2) We've ran into some reliability issues with the network of brokers while using AMQ 5.8.0
(transient) so we resorted to a single beefed up server.
3) Now we are on 5.10.0 and it is able to keep up with the current load but the load tends
to grow and we would want to have some failover capability too.
4) I am running a load test for the same configuration as in production with 5, 7, 6 servers
and again a beefy AMQ 5.10.0 server works fine.
5) When I add another AMQ server using static connector everything seems to work fine under
lighter load. As load grows, even before the message consumption starts falling behind I am
seeing a flood of the "duplicate" messages. They of course are not a problem by themselves
but the DLQ fills up really quickly and AMQs die horrible out of memory death. Exactly the
same behavior under the same setup and load is observed for the 5 or .11-SNAPSHOT (I basically
swap the activemq link in the /opt folder)
6) Keeping the whole system the same, when I switch to 5.9.0 the issue disappears - no "duplicate"
in the log, DLQ does not get populated.

So the question is what gives? The consumers seem to process all the messages at least it
shouldn't be different from 5.9 to 5.10 to 5.11. Do you believe something has changed in 5.10
and it might be a legit behavior?

Let me know. I can provide additional details and assist with debugging if needed

Thank you,

Alex



> "cursor got duplicate" error after upgrade
> ------------------------------------------
>
>                 Key: AMQ-5249
>                 URL: https://issues.apache.org/jira/browse/AMQ-5249
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.9.1, 5.10.0
>            Reporter: Rural Hunter
>
> I was using 5.9.0 and meet one problem so I tried to upgrade activemq. I tried both 5.9.1
and 5.10.0 and encouterred a same problem. I saw messages filled DLQ very quickly. I checked
the clients both producer and consumer but there was no error. I checked activemq log and
found the log is full of these warnings:
> 2014-06-27 23:22:09,337 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.cyyun.webmon.spider.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
- cursor got duplicate: ID:211.com-52399-1400732399425-1:1:235992:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor
| ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,337 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
- cursor got duplicate: ID:nbzjjf22805-34129-1403880308671-1:1:28:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor
| ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,338 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
- cursor got duplicate: ID:jxncxnj2-48598-1403856107346-1:1:6007:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor
| ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,338 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
- cursor got duplicate: ID:jxnc17-60227-1400730816361-1:1:149072:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor
| ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,339 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
- cursor got duplicate: ID:cyyun-46954-1403800808565-1:1:9765:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor
| ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,339 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
- cursor got duplicate: ID:ubuntu-55495-1403497638437-1:1:53086:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor
| ActiveMQ Broker[localhost] Scheduler
> 2014-06-27 23:22:09,340 | WARN  | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@19117501:com.xxxx.update,batchResetNeeded=false,storeHasMessages=true,size=211,cacheEnabled=true,maxBatchSize:200,hasSpace:true
- cursor got duplicate: ID:cyyun-39030-1403880008363-1:1:70:1:1, 4 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor
| ActiveMQ Broker[localhost] Scheduler
> The problem mostly happens right after activemq starts and sometimes happened after activemq
worked normally for a while.
> For now I have to roll back to 5.9.0 and the problem doesn't occure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message