activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ron Koerner (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMQ-3993) NetworkBridge sometimes stops trying to reconnect after connection is lost
Date Mon, 27 Aug 2012 10:53:07 GMT
Ron Koerner created AMQ-3993:
--------------------------------

             Summary: NetworkBridge sometimes stops trying to reconnect after connection is
lost
                 Key: AMQ-3993
                 URL: https://issues.apache.org/jira/browse/AMQ-3993
             Project: ActiveMQ
          Issue Type: Bug
    Affects Versions: 5.6.0
         Environment: using static:// networkConnector (i.e. SimpleDiscoveryAgent)
            Reporter: Ron Koerner
         Attachments: reconnect-problem-annotated.txt

After losing connection due to shutdown of the peer the broker tries to rebuild the connection
once, fails again and stops trying afterwards.

While this also happens with a standard setup, it seems to happen much more often with a certain
type of firewall which always accepts a connection, but closes it if the real destination
cannot be reached.

This can be simulated by using a "socat" forwarder between the two brokers.

The problems seems to lie in the following sequence of events, a race condition and the use
of {{event.failed}} in {{SimpleDiscoveryAgent.serviceFailed}} and {{bridges}} in {{DiscoveryNetworkConnector}}:

# connection "failure" due to ShutdownInfo
#- event.failed=true
#- bridge is unregistered
# start establishing a new connection
#- event.failed=false
#- bridge is not yet registered
# second connection failure of the old connection due to EOF
#- not blocked, since event.failed==false
#- event.failed=true
#- bridge would be unregistered, but currently there is none
#- wait one second (continued below)
# new connection is started
#- bridge is registered
# receive multiple connection failures of the new connection
#- all blocked, since event.failed=true
# continue after one second, try to establish a new connection
#- blocked, since bridge is already registered

To fix this problem a NetworkBridge should probably not be allowed to call {{SimpleDiscoveryAgent.serviceFailed}}
more than once, since {{event.failed}} cannot keep track of multiple connections at one time.

The chain of events holds a lot of race conditions. If the second failure of the old connection
occurs before the new connection is started (which seems to be the case most of the time)
or the new connection's bridge is registered before the EOF occurs, the problem does not manifest.

Attached is a log excerpt with my comments about the state of event.failed and bridges.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message