activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jnicolay <j.nico...@topdesk.com>
Subject problems getting test results deterministic
Date Wed, 20 Jul 2016 15:15:31 GMT
Hi all

We ran into some strange behavior in our application and I tried to reproduce 
it in a unit test.

Our setup looks somewhat like this:
+-----------+       +--------+      +-----------+
| embedded1 | <---> | master | <--> | embedded2 |
+-----------+       +--------+      +-----------+
        \------  ----/     \---  ------/
               \/              \/
+-----------+   |   +--------+  |   +-----------+
| producer1 |---+   | slave  |  +---| producer2 |
+-----------+   |   +--------+  |   +-----------+
                |               | 
+-------------+ |               | +-------------+
| subscriber1 |-+               +-| subscriber2 |
+-------------+                   +-------------+


We have a master/slave broker pair in the middle and two embedded brokers 
called embedded-client1 and embedded-client2 that connect to it with 
"static:failover://(tcp://localhost:<masterPort>,tcp://localhost:<slavePort>)?randomize=false&maxReconnectAttempts=0

Then we're simulating two clients, one uses the connection URL 
"failover://(tcp://localhost:<masterPort>",tcp://localhost:<slavePort>,vm://embedded-
client1)?randomize=false" for sending and subscribing,
the second uses the connection URL 
"failover://(tcp://localhost:<masterPort>",tcp://localhost:<slavePort>,vm://embedded-
client2)?randomize=false" for sending and subscribing.

Both clients register a durable subscriber and after both have registered (and 
we gave the subscriptions some time to propagate) both clients send a message.

After waiting again some time so the message is delivered we kill the master 
and every client sends another message.
At this point I expect the producers to notice that they can neither connect 
to the master, nor the slave (because it hasn't had time to start yet) and 
falls back to delivering the message to their embedded brokers.
Also I expect the subscribers to notice their connection is gone and subscribe 
at their embedded brokers.

So given enough time, both clients should have received the first two messages 
and the second one they sent.

Then the slave starts up (in our test we manually start it at this point) and 
given enough time, both embedded brokers should reestablish their network 
bridge, forward the message that was delivered to them to the central broker 
and this should forward the messages to the other embedded broker with it's 
subscriber.

So both subscribers should have received all four messages. (In different 
order, but that's ok.)

What actually happens is: The test works. once. But when I run it 20 times, it 
looks like this:
- The first time always works.
- A statistical average of 6 times it succeeds.
- A statistical average of 4 times I get a JMSException that a client id is 
already registered.
- A statistical average of 2 times the clients don't get their own messages 
after the central broker died.
- A statistical average of 8 times the clients already received all 4 messages 
before the central broker was restarted.
- Even though all brokers have been told to use a temporary directory to store 
their data that's wiped after every test run (junit TemporaryFolder rule), 
directories "activemq-data/<name-of-embedded-broker1>", "activemq-data/<name-
of-embedded-broker2>" are created in the current directory.

The test case already stops all 4 brokers that are started after the test with 
stop() and waitUntilStopped().


I have the impression that sometimes the connection does not go to the 
embedded broker that I manually created but a new embedded broker with default 
settings is created instead. But I have no clue why this happens sometimes. 
Maybe the embedded instances from the last test run aren't disposed properly 
yet. But then again I have no clue how to clean the state between test runs 
properly.

At the moment the test starts with a 4 second delay, without it the number of 
JMSExceptions rises dramatically.


I'll attach a small maven project with this test case, as well as a log file 
of a test run where all 4 test outcomes happened.


Any help would be greatly appreciated.

Thanks
Joachim


maven-test-project.zip (9K) <http://activemq.2283324.n4.nabble.com/attachment/4714185/0/maven-test-project.zip>
tests.log.zip (904K) <http://activemq.2283324.n4.nabble.com/attachment/4714185/1/tests.log.zip>




--
View this message in context: http://activemq.2283324.n4.nabble.com/problems-getting-test-results-deterministic-tp4714185.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message