activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQ-2774) Network of brokers : Multicast discovery stopped to work
Date Tue, 06 Jul 2010 09:35:53 GMT

    [ https://issues.apache.org/activemq/browse/AMQ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=60448#action_60448
] 

Eric commented on AMQ-2774:
---------------------------

I'm sure that I identified a Latch problem in Multicast Network Discovery mechanism on Duplex
connection 

The multicast notifier thread is blocked. here the trace (on Fuse-5.3.0.5)

"Notifier-MulticastDiscoveryAgent-listener:DiscoveryNetworkConnector:NOCSupervisorP5-ADMIN-OUT-IN:BrokerService[SIBBusModule-NOCP5-tpnocp08s-bus]"
daemon prio=10 tid=0x0000000044ff2400 nid=0x1389 waiting on condition [0x0000000044c26000..0x0000000044c26b90]

   java.lang.Thread.State: WAITING (parking) 
        at sun.misc.Unsafe.park(Native Method) 
        - parking to wait for  <0x00002aaab3dd66f0> (a java.util.concurrent.CountDownLatch$Sync)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) 
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)

        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) 
        at org.apache.activemq.network.DemandForwardingBridgeSupport.start(DemandForwardingBridgeSupport.java:231)

        at org.apache.activemq.network.DiscoveryNetworkConnector.onServiceAdd(DiscoveryNetworkConnector.java:114)

        at org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent$2.run(MulticastDiscoveryAgent.java:484)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:619) 

The problem appears when the network is quickly and alternatively on/off between the two components.

The bridge is created in one direction, but the answer can not be received. 

The thread is blocked on the CountDownLatch. Even if multicast frames are received, the component
can not establish a new network connection. 

Here are an corresponding activemq trace 

When it is OK : 
2010-06-22 22:56:24,500 [-tpnocp08s-bus]] INFO  DiscoveryNetworkConnector      - Establishing
network connection from vm://SIBBusModule-NOCP5-tpnocp08s-bus to tcp://tpnocp11v-bus.vdm.priv.amm.noc:14101?useLocalHost=false

2010-06-22 22:56:26,083 [nocp08s-bus#160] INFO  DemandForwardingBridge         - Network connection
between vm://SIBBusModule-NOCP5-tpnocp08s-bus#160 and tcp://tpnocp11v-bus.vdm.priv.amm.noc/10.18.126.30:14101(SIBBusSupervisor-tpnocp11v-bus)
has been established. 

2010-06-22 22:57:34,807 [-tpnocp08s-bus]] INFO  DemandForwardingBridge         - SIBBusModule-NOCP5-tpnocp08s-bus
bridge to SIBBusSupervisor-tpnocp11v-bus stopped 

2010-06-22 22:57:34,811 [-tpnocp08s-bus]] INFO  DiscoveryNetworkConnector      - Establishing
network connection from vm://SIBBusModule-NOCP5-tpnocp08s-bus to tcp://tpnocp11v-bus.vdm.priv.amm.noc:14101?useLocalHost=false

2010-06-22 22:57:39,064 [nocp08s-bus#162] INFO  DemandForwardingBridge         - Network connection
between vm://SIBBusModule-NOCP5-tpnocp08s-bus#162 and tcp://tpnocp11v-bus.vdm.priv.amm.noc/10.18.126.30:14101(SIBBusSupervisor-tpnocp11v-bus)
has been established. 

2010-06-22 22:58:42,578 [-tpnocp08s-bus]] INFO  DemandForwardingBridge         - SIBBusModule-NOCP5-tpnocp08s-bus
bridge to SIBBusSupervisor-tpnocp11v-bus stopped 

When it is KO : "Unknown" 

2010-06-22 22:58:42,648 [-tpnocp08s-bus]] INFO  DiscoveryNetworkConnector      - Establishing
network connection from vm://SIBBusModule-NOCP5-tpnocp08s-bus to tcp://tpnocp11v-bus.vdm.priv.amm.noc:14101?useLocalHost=false

2010-06-22 22:59:18,031 [18.126.30:14101] WARN  DemandForwardingBridge         - Network connection
between vm://SIBBusModule-NOCP5-tpnocp08s-bus#164 and tcp://tpnocp11v-bus.vdm.priv.amm.noc/10.18.126.30:14101
shutdown due to a remote error: java.net.SocketException: Connection reset 
2010-06-22 22:59:18,033 [NetworkBridge  ] INFO  DemandForwardingBridge         - SIBBusModule-NOCP5-tpnocp08s-bus
bridge to Unknown stopped 


Here is the other side corresponding activemq trace 

activemq-server.log:2010-06-22 22:55:44,295 [26.190.27:40517] INFO  TransportConnection  
         - Created Duplex Bridge back to SIBBusModule-NOCP5-tpnocp08s-bus 

activemq-server.log:2010-06-22 22:56:24,438 [26.190.27:40517] INFO  DemandForwardingBridge
        - SIBBusSupervisor-tpnocp11v-bus bridge to SIBBusModule-NOCP5-tpnocp08s-bus stopped


activemq-server.log:2010-06-22 22:56:26,135 [26.190.27:40518] INFO  TransportConnection  
         - Created Duplex Bridge back to SIBBusModule-NOCP5-tpnocp08s-bus 
activemq-server.log:2010-06-22 22:56:26,135 [ocp11v-bus#1770] INFO  DemandForwardingBridge
        - Network connection between vm://SIBBusSupervisor-tpnocp11v-bus#1770 and tcp:///10.26.190.27:40518(SIBBusModule-NOCP5-tpnocp08s-bus)
has been established. 

activemq-server.log:2010-06-22 22:57:34,818 [26.190.27:40518] INFO  DemandForwardingBridge
        - SIBBusSupervisor-tpnocp11v-bus bridge to SIBBusModule-NOCP5-tpnocp08s-bus stopped


activemq-server.log:2010-06-22 22:57:39,153 [26.190.27:40519] INFO  TransportConnection  
         - Created Duplex Bridge back to SIBBusModule-NOCP5-tpnocp08s-bus 
activemq-server.log:2010-06-22 22:57:39,153 [ocp11v-bus#1806] INFO  DemandForwardingBridge
        - Network connection between vm://SIBBusSupervisor-tpnocp11v-bus#1806 and tcp:///10.26.190.27:40519(SIBBusModule-NOCP5-tpnocp08s-bus)
has been established. 

activemq-server.log:2010-06-22 22:58:44,328 [26.190.27:40519] INFO  DemandForwardingBridge
        - SIBBusSupervisor-tpnocp11v-bus bridge to SIBBusModule-NOCP5-tpnocp08s-bus stopped



> Network of brokers : Multicast discovery stopped to work
> --------------------------------------------------------
>
>                 Key: AMQ-2774
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2774
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.2.0
>         Environment: Linux
>            Reporter: Eric
>             Fix For: 5.4.1
>
>         Attachments: JMAC-BEA-lastlog.log-20100315
>
>
> Hi everybody
> I experiment a big problem with the multicast discovery algorithm, in a network of brokers
topology.
> In some conditions, a broker can't reestablish a distant connection even if the distant
broker is restarted.
> I have the log traces that would help to identify the origin of the problem.
> When there is no discovery/connection error, I can see these 2 lines in the activemq
log file
> #08 Jun 2010 14:31:30,639  INFO  [Multicast Discovery Agent Notifier] org.apache.activemq.network.DiscoveryNetworkConnector
> Establishing network connection between from vm://ACCLU-tpnocp04v to tcp://tpnocp09v-bus:13100?useLocalHost=false
> #08 Jun 2010 14:31:30,692  INFO  [StartLocalBridge: localBroker=vm://ACCLU-tpnocp04v#26]
org.apache.activemq.network.DemandForwardingBridge
> Network connection between vm://ACCLU-tpnocp04v#26 and tcp://tpnocp09v-bus/10.18.126.28:13100(MOM-tpnocp09v)
has been established.
> When the connection is broken, I can see this line in the log.
> #11 Jun 2010 12:37:32,585  INFO  [Multicast Discovery Agent Notifier] org.apache.activemq.network.DemandForwardingBridge
> ACCLU-tpnocp04v bridge to MOM-tpnocp09v stopped
> Then the current ACCLU-tpnocp04v broker tries to reestablish the connection :
> #11 Jun 2010 12:37:34,475  INFO  [Multicast Discovery Agent Notifier] org.apache.activemq.network.DiscoveryNetworkConnector
> Establishing network connection between from vm://ACCLU-tpnocp04v to tcp://tpnocp09v-bus:13100?useLocalHost=false
> But, here, the second line of the log ("has been established") doesn't appear in the
log file !! I don't know exactly if the connection is up or not.
> Then the connection is broken again (look at "Unknown" instead of "MOM-tpnocp09v".
> #11 Jun 2010 13:33:58,655  WARN  [ActiveMQ Transport: tcp://tpnocp09v-bus/10.18.126.28:13100]
org.apache.activemq.network.DemandForwardingBridge
> Network connection between vm://ACCLU-tpnocp04v#58 and tcp://tpnocp09v-bus/10.18.126.28:13100
shutdown due to a remote error: java.net.SocketException: Connection reset
> #11 Jun 2010 13:33:58,657  INFO  [NetworkBridge] org.apache.activemq.network.DemandForwardingBridge^M
> ACCLU-tpnocp04v bridge to Unknown stopped
> And, now, even if I restart the distant broker ( MOM-tpnocp09v ), no line (Establishing/Has
been established) appears, and no network connection is reestablished between ACCLU-tpnocp04v
and MOM-tpnocp09v. it seems that this ACCLU-tpnocp04v broker can no longer establish a connection
with the MOM-tpnocp09v broker !!!
> The production teams tell me that this problem seems not to be resolved in fuse-5.3.0.6
version.
> Eric-AWL

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message