activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Gellings (JIRA)" <jira+amq...@apache.org>
Subject [jira] Created: (AMQNET-252) stress tests not showing all messages being processed as predicted
Date Thu, 20 May 2010 20:42:51 GMT
stress tests not showing all messages being processed as predicted
------------------------------------------------------------------

                 Key: AMQNET-252
                 URL: https://issues.apache.org/activemq/browse/AMQNET-252
             Project: ActiveMQ .Net
          Issue Type: Bug
          Components: NMS
    Affects Versions: 1.3.0
         Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers
in windows services.  producers sending messages from Windows XP
            Reporter: Mark Gellings
            Assignee: Jim Gomes
             Fix For: 1.3.0


Should say right away the below is the stress test model we used when thoroughly testing NMS
v1.1 (of which we use quite extensively in our production environment).  The tests pass consistently
with NMS v1.1.  These tests are done using a framework we wrote that rests on top of Apache
NMS ActiveMQ provider.  We can't distribute that code for proprietary reasons.  I did however
include a console application to be used of which I have replicated the problem on at least
one occasion, albeit the stress test is not near as sophisticated.

Stress Test model

// *note first character "P" is a producer
// *note first character "C" is a topic consumer
// *note first character "v" is a virtual topic
// *note character "t" is a topic
// *note all numbers denote seperate topics, producers, consumers, and brokers
// *note (durable) means a durable subscription

                 |           |
P1 --- vt1 ----> |           | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER
1 
P2 ---- t1 ----> |           |
P3 ---- t1 ----> |  BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 --> 
P4 ---- t2 ----> |           |                                                BROKER 2
 --- t3 (durable), t4 ---> C4 
P5 ---- t2 ----> |           | -------- t1, t2 ------> C3 -------- t4 ------>
P6 ---- t5 ----> |           |
                 |           | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER
1
                 

Stress Test #1

* First producer sends 10,000 msgs to VirtualTopic.t1
* Two producers send 10,000 msgs to t1.
* Two producers send 10,000 msgs to t2.
* One producer sends 10,000 msgs to t5.

With the above test all brokers, producers, and consumers were not restarted.

Test passes, results:

ProcessorName           msg_cnt
QuadNMSConsumer/vt1     10000		
QuadNMSConsumer/t11	20000
QuadNMSConsumer/t12	20000
QuadNMSConsumer/t13	20000
QuadNMSConsumer/t23	60000
QuadNMSConsumer/t34	30000
QuadNMSConsumer/t44	80000
QuadNMSConsumer/t52	10000
QuadNMSConsumer/t55	10000


Stress Test #2 

* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.

# Consumers use client acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure
they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave
setup) at least once to ensure that we do not receive duplicate messages

The end results show (from what I can tell) a failed test according to vt1, t12, t52, and
t55.  The other topic subscriptions are non-durable so the restarts throw off the counts.

ProcessorName           msg_cnt		note
QuadNMSConsumer/vt1	990		should be 1000 messages
QuadNMSConsumer/t11	860		should be 2000 messages
QuadNMSConsumer/t12	822		should be 2000 messages
QuadNMSConsumer/t13	185
QuadNMSConsumer/t23	1802
QuadNMSConsumer/t34	1270
QuadNMSConsumer/t44	1987
QuadNMSConsumer/t52	448		should be 1000 messages
QuadNMSConsumer/t55	479		should be 1000 messages

Stress Test #3


* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.

# Consumers use individual acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure
they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave
setup) at least once to ensure that we do not receive duplicate messages, as the consumers
use our home grown idempotent consumer logic

The end results show another failed test.

ProcessorName           msg_cnt		note
QuadNMSConsumer/vt1	995		should be 1000 messages
QuadNMSConsumer/t11	2000		should be 2000 messages
QuadNMSConsumer/t12	1995		should be 2000 messages
QuadNMSConsumer/t13	329
QuadNMSConsumer/t23	3028
QuadNMSConsumer/t34	2988
QuadNMSConsumer/t44	3356
QuadNMSConsumer/t52	995		should be 1000 messages
QuadNMSConsumer/t55	995  		should be 1000 messages

Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently.
I can see the message in the database sitting in the activemq messages table with the failed
stress tests. 

These stress tests are rather large so I put together a console application to replicate the
problem with a virtual topic.

Putting this sort of stress test in a unit test probably wouldn't make sense which is why
I've attached the console with test case.  We'll want to figure out the problem and then write
a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.

I should also note I can't replicate this problem easily without our framework on top, but
have replicated the problem with the test below at least once out of about 5 tests.

Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.

To run the test:

1) start a broker somewhere
2) run three instances of the attached console.
3) start two consumers processing a virtual topic
4) start one producer publishing 20,000 messages to the virtual topic
5) Press enter on the console consumers to restart them and restart the broker a number of
times
7) When all messages are processed by the consoles, look at the queue in the web console.
 There will be at least one pending message on the virtual topic but when you drill down the
message doesn't exist.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message