activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Gomes (JIRA)" <jira+amq...@apache.org>
Subject [jira] Commented: (AMQNET-252) stress tests not showing all messages being processed as predicted
Date Tue, 10 Aug 2010 17:39:48 GMT

    [ https://issues.apache.org/activemq/browse/AMQNET-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=61141#action_61141
] 

Jim Gomes commented on AMQNET-252:
----------------------------------

I've been investigating the problem report, and I have played around with the contributed
test case code.  After some clean-up to the code, I fail to see where the problem lies.  The
issue write-up does not succinctly describe a problem.  It has too much extra stuff in it.
 As near as I can tell from the report, there seems to be a problem with the web-console of
ActiveMQ showing an extra phantom pending message.

Peeking behind ActiveMQ Broker's curtain and looking directly into its data store and seeing
messages still in there doesn't necessarily constitute a bug.  What is important is whether
messages get delivered correctly, and whether the message receipt is acknowledged correctly.

If there is a bug here that shows a failure in behavior, and not just a difference in behavior
from NMS 1.1 to 1.3, then please write up a new JIRA issue clearly describing the scenario
that shows actual results of the test, as well as the expected results of the test.  I am
interested in ensuring the quality and reliability of NMS, but there needs to be a clearer
instructions on how to reproduce and diagnose the problem.


> stress tests not showing all messages being processed as predicted
> ------------------------------------------------------------------
>
>                 Key: AMQNET-252
>                 URL: https://issues.apache.org/activemq/browse/AMQNET-252
>             Project: ActiveMQ .Net
>          Issue Type: Bug
>          Components: NMS
>    Affects Versions: 1.3.0
>         Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers
in windows services.  producers sending messages from Windows XP
>            Reporter: Mark Gellings
>            Assignee: Jim Gomes
>             Fix For: 1.4.0
>
>         Attachments: NativeNMSConsumerAndProducer.zip
>
>
> Should say right away the below is the stress test model we used when thoroughly testing
NMS v1.1 (of which we use quite extensively in our production environment).  The tests pass
consistently with NMS v1.1.  These tests are done using a framework we wrote that rests on
top of Apache NMS ActiveMQ provider.  We can't distribute that code for proprietary reasons.
 I did however include a console application to be used of which I have replicated the problem
on at least one occasion, albeit the stress test is not near as sophisticated.
> Stress Test model
> // *note first character "P" is a producer
> // *note first character "C" is a topic consumer
> // *note first character "v" is a virtual topic
> // *note character "t" is a topic
> // *note all numbers denote seperate topics, producers, consumers, and brokers
> // *note (durable) means a durable subscription
> {noformat}        
>                  |                       |
> P1 --- vt1 ----> |                       | ---- t1 (durable), vt1 ----> C1 ----
t2 ----> BROKER 1 
> P2 ---- t1 ----> |                       |
> P3 ---- t1 ----> |      BROKER 1         | -- t1 (durable), t5 (durable) --> C2
-- t3 -------> 
> P4 ---- t2 ----> |                       |                                       
                BROKER 2  --- t3 (durable), t4 ---> C4 
> P5 ---- t2 ----> |                       | --------------- t1, t2 ------> C3 --------
t4 ------>
> P6 ---- t5 ----> |                       |
>                  |                       | ------- vt1, t5 (durable) -----> C5 ----
t2 ----> BROKER 1{noformat}
>               
> Stress Test #1
> * First producer sends 10,000 msgs to VirtualTopic.t1
> * Two producers send 10,000 msgs to t1.
> * Two producers send 10,000 msgs to t2.
> * One producer sends 10,000 msgs to t5.
> With the above test all brokers, producers, and consumers were not restarted.
> Test passes, results:
> ProcessorName           msg_cnt
> QuadNMSConsumer/vt1     10000		
> QuadNMSConsumer/t11	20000
> QuadNMSConsumer/t12	20000
> QuadNMSConsumer/t13	20000
> QuadNMSConsumer/t23	60000
> QuadNMSConsumer/t34	30000
> QuadNMSConsumer/t44	80000
> QuadNMSConsumer/t52	10000
> QuadNMSConsumer/t55	10000
> Stress Test #2 
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use client acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to
ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave
setup) at least once to ensure that we do not receive duplicate messages
> The end results show (from what I can tell) a failed test according to vt1, t12, t52,
and t55.  The other topic subscriptions are non-durable so the restarts throw off the counts.
> ProcessorName           msg_cnt		note
> QuadNMSConsumer/vt1	990		should be 1000 messages
> QuadNMSConsumer/t11	860		should be 2000 messages
> QuadNMSConsumer/t12	822		should be 2000 messages
> QuadNMSConsumer/t13	185
> QuadNMSConsumer/t23	1802
> QuadNMSConsumer/t34	1270
> QuadNMSConsumer/t44	1987
> QuadNMSConsumer/t52	448		should be 1000 messages
> QuadNMSConsumer/t55	479		should be 1000 messages
> Stress Test #3
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use individual acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to
ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave
setup) at least once to ensure that we do not receive duplicate messages, as the consumers
use our home grown idempotent consumer logic
> The end results show another failed test.
> ProcessorName           msg_cnt		note
> QuadNMSConsumer/vt1	995		should be 1000 messages
> QuadNMSConsumer/t11	2000		should be 2000 messages
> QuadNMSConsumer/t12	1995		should be 2000 messages
> QuadNMSConsumer/t13	329
> QuadNMSConsumer/t23	3028
> QuadNMSConsumer/t34	2988
> QuadNMSConsumer/t44	3356
> QuadNMSConsumer/t52	995		should be 1000 messages
> QuadNMSConsumer/t55	995  		should be 1000 messages
> Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently.
I can see the message in the database sitting in the activemq messages table with the failed
stress tests. 
> These stress tests are rather large so I put together a console application to replicate
the problem with a virtual topic.
> Putting this sort of stress test in a unit test probably wouldn't make sense which is
why I've attached the console with test case.  We'll want to figure out the problem and then
write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test
passes.
> I should also note I can't replicate this problem easily without our framework on top,
but have replicated the problem with the test below at least once out of about 5 tests.
> Only thing I really have to go by at this point is the above stress tests pass with NMS
v1.1.
> To run the test:
> 1) start a broker somewhere
> 2) run three instances of the attached console.
> 3) start two consumers processing a virtual topic
> 4) start one producer publishing 20,000 messages to the virtual topic
> 5) Press enter on the console consumers to restart them and restart the broker a number
of times
> 7) When all messages are processed by the consoles, look at the queue in the web console.
 There will be at least one pending message on the virtual topic but when you drill down the
message doesn't exist.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message