qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Moravec <pmora...@redhat.com>
Subject Re: network interface down and up, cluster start failed
Date Wed, 25 Jul 2012 11:04:09 GMT
Hi,
what is the purpose of shutting down network interface for few minutes? Just a test simulating
a network failure? If so, then I recommend using either of these methods that simulate so
more closely (and no issues should occur):

   1. Pull the network cable that is used for cluster communication on one of the cluster
nodes.
   2. Bring down the corresponding port on the switch.
   3. Use iptables to drop both incoming and outgoing traffic for the IP associated with heartbeat
traffic for the cluster node. There is a script on the the Github.com repository for corosync
that be used: https://github.com/corosync/corosync/blob/master/cts/agents/net_breaker.sh
$ ~/bin/net_breaker BreakCommCmd 192.168.1.101

Kind regards,
Pavel


----- Original Message -----
> From: "sun198507" <tianqingsun120@gmail.com>
> To: users@qpid.apache.org
> Sent: Tuesday, July 24, 2012 10:04:22 AM
> Subject: network interface down and up, cluster start failed
> 
> Dear All!
> I am testing cluster of cpp qpid. Testing cluster only include a
> node.Qpid
> is 0.14 release,openais is 0.8.0.3.When network is ok, everything is
> ok.But
> When I down the network interface , wait for a few minutes and up the
> network interface, the cluster node start failed for "critical
> Unexpected
> error: Daemon startup failed: Cannot join CPG group msgplat: try
> again (6)".
> Restart openais, Qpid start success. More detailed failed messages as
> follows. Anybody  can tell me why the case happen or give some ideas?
> 
> =================================================
> Jul 21 13:53:29 test kernel: ADDRCONF(NETDEV_UP): eth0: link is not
> ready
> Jul 21 13:53:29 test openais[2222]: [TOTEM] Receive multicast socket
> recv
> buffer size (262142 bytes).
> Jul 21 13:53:29 test openais[2222]: [TOTEM] Transmit multicast socket
> send
> buffer size (262142 bytes).
> Jul 21 13:53:29 test openais[2222]: [TOTEM] The network interface is
> down.
> Jul 21 13:53:29 test openais[2222]: [TOTEM] entering GATHER state
> from 15.
> Jul 21 13:53:30 test openais[2222]: [TOTEM] entering GATHER state
> from 0.
> Jul 21 13:53:30 test openais[2222]: [TOTEM] Creating commit token
> because I
> am the rep.
> Jul 21 13:53:30 test openais[2222]: [TOTEM] Saving state aru 8b4d06
> high seq
> received 8b4d06
> Jul 21 13:53:30 test openais[2222]: [TOTEM] entering COMMIT state.
> Jul 21 13:53:30 test openais[2222]: [TOTEM] entering RECOVERY state.
> Jul 21 13:53:30 test openais[2222]: [TOTEM] position [0] member
> 127.0.0.1:
> Jul 21 13:53:30 test openais[2222]: [TOTEM] previous ring seq 100 rep
> 206.122.131.141
> Jul 21 13:53:30 test openais[2222]: [TOTEM] aru 8b4d06 high delivered
> 8b4d06
> received flag 0
> Jul 21 13:53:30 test openais[2222]: [TOTEM] Did not need to originate
> any
> messages in recovery.
> Jul 21 13:53:30 test openais[2222]: [TOTEM] Storing new sequence id
> for ring
> 68
> Jul 21 13:53:30 test openais[2222]: [TOTEM] Sending initial ORF token
> Jul 21 13:53:30 test openais[2222]: [CLM  ] CLM CONFIGURATION CHANGE
> Jul 21 13:53:30 test openais[2222]: [CLM  ] New Configuration:
> Jul 21 13:53:30 test openais[2222]: [CLM  ] #011r(0) ip(127.0.0.1)
> Jul 21 13:53:30 test openais[2222]: [CLM  ] Members Left:
> Jul 21 13:53:30 test openais[2222]: [CLM  ] Members Joined:
> Jul 21 13:53:30 test openais[2222]: [SYNC ] This node is within the
> primary
> component and will provide service.
> Jul 21 13:53:30 test openais[2222]: [CLM  ] CLM CONFIGURATION CHANGE
> Jul 21 13:53:30 test openais[2222]: [CLM  ] New Configuration:
> Jul 21 13:53:30 test openais[2222]: [CLM  ] #011r(0) ip(127.0.0.1)
> Jul 21 13:53:30 test openais[2222]: [CLM  ] Members Left:
> Jul 21 13:53:30 test openais[2222]: [CLM  ] Members Joined:
> Jul 21 13:53:30 test openais[2222]: [SYNC ] This node is within the
> primary
> component and will provide service.
> Jul 21 13:53:30 test openais[2222]: [TOTEM] entering OPERATIONAL
> state.
> Jul 21 13:53:30 test openais[2222]: [TOTEM] Message continuation
> doesn't
> match previous frag e: 0 - a: 146
> Jul 21 13:53:30 test openais[2222]: [TOTEM] Throwing away broken
> message:
> continuation 0, index 0
> Jul 21 13:53:30 test openais[2222]: [CLM  ] got nodejoin message
> 127.0.0.1
> Jul 21 13:53:32 test kernel: igb: eth0 NIC Link is Up 1000 Mbps Full
> Duplex,
> Flow Control: RX
> Jul 21 13:53:32 test kernel: ADDRCONF(NETDEV_CHANGE): eth0: link
> becomes
> ready
> Jul 21 13:53:33 test MsgBroker[2263]: 2012-07-21 13:53:33 critical
> Error
> delivering frames: Unknown connection: Frame[BEbe; channel=1;
> {QueueQueryBody: queue=1ClueVersion; }] data 127.0.0.1:2263-124293
> read-credit=1 (qpid/cluster/Cluster.cpp:544)
> Jul 21 13:53:33 test MsgBroker[2263]: 2012-07-21 13:53:33 notice
> cluster(206.122.131.141:2263 LEFT) leaving cluster msgplat
> Jul 21 13:53:33 test MsgBroker[2263]: 2012-07-21 13:53:33 notice Shut
> down
> Jul 21 13:53:34 test openais[2222]: [TOTEM] Receive multicast socket
> recv
> buffer size (262142 bytes).
> Jul 21 13:53:34 test openais[2222]: [TOTEM] Transmit multicast socket
> send
> buffer size (262142 bytes).
> Jul 21 13:53:34 test openais[2222]: [TOTEM] The network interface
> [206.122.131.141] is now up.
> Jul 21 13:53:34 test openais[2222]: [TOTEM] entering GATHER state
> from 15.
> Jul 21 13:53:35 test openais[2222]: [TOTEM] entering GATHER state
> from 0.
> Jul 21 13:53:35 test openais[2222]: [TOTEM] Creating commit token
> because I
> am the rep.
> Jul 21 13:53:35 test openais[2222]: [TOTEM] Saving state aru 18a high
> seq
> received 18a
> Jul 21 13:53:35 test openais[2222]: [TOTEM] entering COMMIT state.
> Jul 21 13:53:35 test openais[2222]: [TOTEM] entering RECOVERY state.
> Jul 21 13:53:35 test openais[2222]: [TOTEM] position [0] member
> 206.122.131.141:
> Jul 21 13:53:35 test openais[2222]: [TOTEM] previous ring seq 104 rep
> 127.0.0.1
> Jul 21 13:53:35 test openais[2222]: [TOTEM] aru 18a high delivered
> 18a
> received flag 0
> Jul 21 13:53:35 test openais[2222]: [TOTEM] Did not need to originate
> any
> messages in recovery.
> Jul 21 13:53:35 test openais[2222]: [TOTEM] Storing new sequence id
> for ring
> 6c
> Jul 21 13:53:35 test openais[2222]: [TOTEM] Sending initial ORF token
> Jul 21 13:53:35 test openais[2222]: [CLM  ] CLM CONFIGURATION CHANGE
> Jul 21 13:53:35 test openais[2222]: [CLM  ] New Configuration:
> Jul 21 13:53:35 test openais[2222]: [CLM  ] #011r(0)
> ip(206.122.131.141)
> Jul 21 13:53:35 test openais[2222]: [CLM  ] Members Left:
> Jul 21 13:53:35 test openais[2222]: [CLM  ] Members Joined:
> Jul 21 13:53:35 test openais[2222]: [SYNC ] This node is within the
> primary
> component and will provide service.
> Jul 21 13:53:35 test openais[2222]: [CLM  ] CLM CONFIGURATION CHANGE
> Jul 21 13:53:35 test openais[2222]: [CLM  ] New Configuration:
> Jul 21 13:53:35 test openais[2222]: [CLM  ] #011r(0)
> ip(206.122.131.141)
> Jul 21 13:53:35 test openais[2222]: [CLM  ] Members Left:
> Jul 21 13:53:35 test openais[2222]: [CLM  ] Members Joined:
> Jul 21 13:53:35 test openais[2222]: [SYNC ] This node is within the
> primary
> component and will provide service.
> Jul 21 13:53:35 test openais[2222]: [TOTEM] entering OPERATIONAL
> state.
> Jul 21 13:53:48 test MsgBroker[29877]: 2012-07-21 13:53:48 notice
> Starting
> watchdog process with interval of 60 seconds
> Jul 21 13:53:48 test MsgBroker[29877]: 2012-07-21 13:53:48 notice
> Initializing CPG
> Jul 21 13:53:48 test MsgBroker[29877]: 2012-07-21 13:53:48 critical
> Unexpected error: Cannot join CPG group msgplat: try again (6)
> Jul 21 13:53:48 test MsgBroker[29876]: 2012-07-21 13:53:48 critical
> Unexpected error: Daemon startup failed: Cannot join CPG group
> msgplat: try
> again (6)
> Jul 21 13:54:09 test MsgBroker[29932]: 2012-07-21 13:54:09 notice
> Starting
> watchdog process with interval of 60 seconds
> Jul 21 13:54:09 test MsgBroker[29932]: 2012-07-21 13:54:09 notice
> Initializing CPG
> Jul 21 13:54:09 test MsgBroker[29932]: 2012-07-21 13:54:09 critical
> Unexpected error: Cannot join CPG group msgplat: try again (6)
> Jul 21 13:54:09 test MsgBroker[29931]: 2012-07-21 13:54:09 critical
> Unexpected error: Daemon startup failed: Cannot join CPG group
> msgplat: try
> again (6)
> Jul 21 13:54:29 test MsgBroker[29989]: 2012-07-21 13:54:29 notice
> Starting
> watchdog process with interval of 60 seconds
> Jul 21 13:54:29 test MsgBroker[29989]: 2012-07-21 13:54:29 notice
> Initializing CPG
> Jul 21 13:54:29 test MsgBroker[29989]: 2012-07-21 13:54:29 critical
> Unexpected error: Cannot join CPG group msgplat: try again (6)
> Jul 21 13:54:29 test MsgBroker[29988]: 2012-07-21 13:54:29 critical
> Unexpected error: Daemon startup failed: Cannot join CPG group
> msgplat: try
> again (6)
> Jul 21 13:54:49 test MsgBroker[30044]: 2012-07-21 13:54:49 notice
> Starting
> watchdog process with interval of 60 seconds
> Jul 21 13:54:49 test MsgBroker[30044]: 2012-07-21 13:54:49 notice
> Initializing CPG
> 
> ==========================================================
> 
> At the same test with network interface down and up, Qpid Broker hang
> for
> critical Unexpected error: Timed out waiting for daemon (If store
> recovery
> is in progress, use longer wait time), not accept any connection from
> client
> until restart aisexec and Qpid. Detailed failed messages as follows.
> 
> 
> Jul 21 15:16:28 test openais[11845]: [MAIN ] AIS Executive Service
> RELEASE
> 'subrev 1358 version 0.80.3'
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Copyright (C) 2002-2006
> MontaVista Software, Inc and contributors.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Copyright (C) 2006 Red
> Hat,
> Inc.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] AIS Executive Service:
> started
> and ready to provide service.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component
> openais_cpg
> loaded.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service
> handler
> 'openais cluster closed process group service v1.01'
> Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component
> openais_cfg
> loaded.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service
> handler
> 'openais configuration service'
> Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component
> openais_msg
> loaded.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service
> handler
> 'openais message service B.01.01'
> Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component
> openais_lck
> loaded.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service
> handler
> 'openais distributed locking service B.01.01'
> Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component
> openais_evt
> loaded.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service
> handler
> 'openais event service B.01.01'
> Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component
> openais_ckpt
> loaded.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service
> handler
> 'openais checkpoint service B.01.01'
> Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component
> openais_amf
> loaded.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service
> handler
> 'openais availability management framework B.01.01'
> Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component
> openais_clm
> loaded.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service
> handler
> 'openais cluster membership service B.01.01'
> Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component
> openais_evs
> loaded.
> Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service
> handler
> 'openais extended virtual synchrony service'
> Jul 21 15:16:28 test openais[11845]: [TOTEM] Token Timeout (1000 ms)
> retransmit timeout (238 ms)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] token hold (180 ms)
> retransmits
> before loss (4 retrans)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] join (50 ms) send_join
> (0 ms)
> consensus (800 ms) merge (200 ms)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] downcheck (1000 ms) fail
> to
> recv const (50 msgs)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] seqno unchanged const
> (30
> rotations) Maximum network MTU 1500
> Jul 21 15:16:28 test openais[11845]: [TOTEM] window size per rotation
> (50
> messages) maximum messages per rotation (17 messages)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] send threads (0 threads)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] RRP token expired
> timeout (238
> ms)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] RRP token problem
> counter (2000
> ms)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] RRP threshold (10
> problem
> count)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] RRP mode set to none.
> Jul 21 15:16:28 test openais[11845]: [TOTEM]
> heartbeat_failures_allowed (0)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] max_network_delay (50
> ms)
> Jul 21 15:16:28 test openais[11845]: [TOTEM] HeartBeat is Disabled.
> To
> enable set heartbeat_failures_allowed > 0
> Jul 21 15:16:28 test openais[11845]: [TOTEM] Receive multicast socket
> recv
> buffer size (262142 bytes).
> Jul 21 15:16:28 test openais[11845]: [TOTEM] Transmit multicast
> socket send
> buffer size (262142 bytes).
> Jul 21 15:16:28 test openais[11845]: [TOTEM] The network interface
> [206.122.131.141] is now up.
> Jul 21 15:16:28 test openais[11845]: [TOTEM] Created or loaded
> sequence id
> 108.206.122.131.141 for this ring.
> Jul 21 15:16:28 test openais[11845]: [TOTEM] entering GATHER state
> from 15.
> Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service
> handler
> 'openais extended virtual synchrony service'
> Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service
> handler
> 'openais cluster membership service B.01.01'
> Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service
> handler
> 'openais availability management framework B.01.01'
> Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service
> handler
> 'openais checkpoint service B.01.01'
> Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service
> handler
> 'openais event service B.01.01'
> Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service
> handler
> 'openais distributed locking service B.01.01'
> Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service
> handler
> 'openais message service B.01.01'
> Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service
> handler
> 'openais configuration service'
> Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service
> handler
> 'openais cluster closed process group service v1.01'
> Jul 21 15:16:28 test openais[11845]: [SYNC ] Not using a virtual
> synchrony
> filter.
> Jul 21 15:16:28 test openais[11845]: [TOTEM] Creating commit token
> because I
> am the rep.
> Jul 21 15:16:28 test openais[11845]: [TOTEM] Saving state aru 0 high
> seq
> received 0
> Jul 21 15:16:28 test openais[11845]: [TOTEM] entering COMMIT state.
> Jul 21 15:16:28 test openais[11845]: [TOTEM] entering RECOVERY state.
> Jul 21 15:16:28 test openais[11845]: [TOTEM] position [0] member
> 206.122.131.141:
> Jul 21 15:16:28 test openais[11845]: [TOTEM] previous ring seq 108
> rep
> 206.122.131.141
> Jul 21 15:16:28 test openais[11845]: [TOTEM] aru 0 high delivered 0
> received
> flag 0
> Jul 21 15:16:28 test openais[11845]: [TOTEM] Did not need to
> originate any
> messages in recovery.
> Jul 21 15:16:28 test openais[11845]: [TOTEM] Storing new sequence id
> for
> ring 70
> Jul 21 15:16:28 test openais[11845]: [TOTEM] Sending initial ORF
> token
> Jul 21 15:16:28 test openais[11845]: [CLM  ] CLM CONFIGURATION CHANGE
> Jul 21 15:16:28 test openais[11845]: [CLM  ] New Configuration:
> Jul 21 15:16:28 test openais[11845]: [CLM  ] Members Left:
> Jul 21 15:16:28 test openais[11845]: [CLM  ] Members Joined:
> Jul 21 15:16:28 test openais[11845]: [SYNC ] This node is within the
> primary
> component and will provide service.
> Jul 21 15:16:28 test openais[11845]: [CLM  ] CLM CONFIGURATION CHANGE
> Jul 21 15:16:28 test openais[11845]: [CLM  ] New Configuration:
> Jul 21 15:16:28 test openais[11845]: [CLM  ] #011r(0)
> ip(206.122.131.141)
> Jul 21 15:16:28 test openais[11845]: [CLM  ] Members Left:
> Jul 21 15:16:28 test openais[11845]: [CLM  ] Members Joined:
> Jul 21 15:16:28 test openais[11845]: [CLM  ] #011r(0)
> ip(206.122.131.141)
> Jul 21 15:16:28 test openais[11845]: [SYNC ] This node is within the
> primary
> component and will provide service.
> Jul 21 15:16:28 test openais[11845]: [TOTEM] entering OPERATIONAL
> state.
> Jul 21 15:16:28 test openais[11845]: [CLM  ] got nodejoin message
> 206.122.131.141
> Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice
> Starting
> watchdog process with interval of 60 seconds
> Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice
> Initializing CPG
> Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice
> cluster(206.122.131.141:11883 PRE_INIT) configuration change:
> 206.122.131.141:11883
> Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice
> cluster(206.122.131.141:11883 PRE_INIT) Members joined:
> 206.122.131.141:11883
> Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice
> SASL
> disabled: No Authentication Performed
> Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice
> Listening
> on TCP/TCP6 port 5672
> Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice
> cluster(206.122.131.141:11883 INIT) cluster-uuid =
> a5d98d15-c518-4de7-8e1a-94dea116844d
> Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice
> cluster(206.122.131.141:11883 READY) joined cluster msgplat
> Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice
> Broker
> running
> Jul 21 15:16:37 test MsgBroker[11883]: 2012-07-21 15:16:37 notice
> Shut down
> Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice
> Starting
> watchdog process with interval of 60 seconds
> Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice
> Initializing CPG
> Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice
> cluster(206.122.131.141:11929 PRE_INIT) configuration change:
> 206.122.131.141:11929
> Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice
> cluster(206.122.131.141:11929 PRE_INIT) Members joined:
> 206.122.131.141:11929
> Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice
> SASL
> disabled: No Authentication Performed
> Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice
> Listening
> on TCP/TCP6 port 5672
> Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice
> cluster(206.122.131.141:11929 INIT) cluster-uuid =
> cd26ef10-d608-42ad-8e37-607f25798127
> Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice
> cluster(206.122.131.141:11929 READY) joined cluster msgplat
> Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice
> Broker
> running
> Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice
> Starting
> watchdog process with interval of 60 seconds
> Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice
> Initializing CPG
> Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice
> cluster(206.122.131.141:11992 PRE_INIT) configuration change:
> 206.122.131.141:11992
> Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice
> cluster(206.122.131.141:11992 PRE_INIT) Members joined:
> 206.122.131.141:11992
> Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice
> SASL
> disabled: No Authentication Performed
> Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice
> Listening
> on TCP/TCP6 port 5672
> Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice
> cluster(206.122.131.141:11992 INIT) cluster-uuid =
> a64ce087-5874-43eb-9e17-0d2aeec5e67f
> Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice
> cluster(206.122.131.141:11992 READY) joined cluster msgplat
> Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice
> Broker
> running
> Jul 21 15:45:02 test openais[11845]: [TOTEM] Receive multicast socket
> recv
> buffer size (262142 bytes).
> Jul 21 15:45:02 test openais[11845]: [TOTEM] Transmit multicast
> socket send
> buffer size (262142 bytes).
> Jul 21 15:45:02 test openais[11845]: [TOTEM] The network interface is
> down.
> Jul 21 15:45:02 test openais[11845]: [TOTEM] entering GATHER state
> from 15.
> Jul 21 15:45:03 test openais[11845]: [TOTEM] entering GATHER state
> from 0.
> Jul 21 15:45:03 test openais[11845]: [TOTEM] Creating commit token
> because I
> am the rep.
> Jul 21 15:45:03 test openais[11845]: [TOTEM] Saving state aru 31b25
> high seq
> received 31b25
> Jul 21 15:45:03 test openais[11845]: [TOTEM] entering COMMIT state.
> Jul 21 15:45:03 test openais[11845]: [TOTEM] entering RECOVERY state.
> Jul 21 15:45:03 test openais[11845]: [TOTEM] position [0] member
> 127.0.0.1:
> Jul 21 15:45:03 test openais[11845]: [TOTEM] previous ring seq 112
> rep
> 206.122.131.141
> Jul 21 15:45:03 test openais[11845]: [TOTEM] aru 31b25 high delivered
> 31b25
> received flag 0
> Jul 21 15:45:03 test openais[11845]: [TOTEM] Did not need to
> originate any
> messages in recovery.
> Jul 21 15:45:03 test openais[11845]: [TOTEM] Storing new sequence id
> for
> ring 74
> Jul 21 15:45:03 test openais[11845]: [TOTEM] Sending initial ORF
> token
> Jul 21 15:45:03 test openais[11845]: [CLM  ] CLM CONFIGURATION CHANGE
> Jul 21 15:45:03 test openais[11845]: [CLM  ] New Configuration:
> Jul 21 15:45:03 test openais[11845]: [CLM  ] #011r(0) ip(127.0.0.1)
> Jul 21 15:45:03 test openais[11845]: [CLM  ] Members Left:
> Jul 21 15:45:03 test openais[11845]: [CLM  ] Members Joined:
> Jul 21 15:45:03 test openais[11845]: [SYNC ] This node is within the
> primary
> component and will provide service.
> Jul 21 15:45:03 test openais[11845]: [CLM  ] CLM CONFIGURATION CHANGE
> Jul 21 15:45:03 test openais[11845]: [CLM  ] New Configuration:
> Jul 21 15:45:03 test openais[11845]: [CLM  ] #011r(0) ip(127.0.0.1)
> Jul 21 15:45:03 test openais[11845]: [CLM  ] Members Left:
> Jul 21 15:45:03 test openais[11845]: [CLM  ] Members Joined:
> Jul 21 15:45:03 test openais[11845]: [SYNC ] This node is within the
> primary
> component and will provide service.
> Jul 21 15:45:03 test openais[11845]: [TOTEM] entering OPERATIONAL
> state.
> Jul 21 15:45:03 test openais[11845]: [CLM  ] got nodejoin message
> 127.0.0.1
> Jul 21 15:45:03 test MsgBroker[11992]: 2012-07-21 15:45:03 critical
> Error
> delivering frames: Unknown connection: Frame[BEbe; channel=1;
> {SessionDetachBody: name=e8b38236-934b-4c2a-99c7-73bc09d91f67; }]
> data
> 127.0.0.1:11992-2488 read-credit=1 (qpid/cluster/Cluster.cpp:544)
> Jul 21 15:45:03 test MsgBroker[11992]: 2012-07-21 15:45:03 notice
> cluster(206.122.131.141:11992 LEFT) leaving cluster msgplat
> Jul 21 15:45:03 test MsgBroker[11992]: 2012-07-21 15:45:03 notice
> Shut down
> Jul 21 15:45:04 test MsgBroker[15998]: 2012-07-21 15:45:04 notice
> Starting
> watchdog process with interval of 60 seconds
> Jul 21 15:45:04 test MsgBroker[15998]: 2012-07-21 15:45:04 notice
> Initializing CPG
> Jul 21 15:55:04 test MsgBroker[15997]: 2012-07-21 15:55:04 critical
> Unexpected error: Timed out waiting for daemon (If store recovery is
> in
> progress, use longer wait time)
> Jul 21 15:59:25 test kernel: ADDRCONF(NETDEV_UP): eth0: link is not
> ready
> Jul 21 15:59:25 test openais[11845]: [TOTEM] Receive multicast socket
> recv
> buffer size (262142 bytes).
> Jul 21 15:59:25 test openais[11845]: [TOTEM] Transmit multicast
> socket send
> buffer size (262142 bytes).
> Jul 21 15:59:25 test openais[11845]: [TOTEM] The network interface
> [206.122.131.141] is now up.
> Jul 21 15:59:25 test openais[11845]: [TOTEM] entering GATHER state
> from 15.
> Jul 21 15:59:26 test openais[11845]: [TOTEM] entering GATHER state
> from 0.
> Jul 21 15:59:26 test openais[11845]: [TOTEM] Creating commit token
> because I
> am the rep.
> Jul 21 15:59:26 test openais[11845]: [TOTEM] Saving state aru 2d high
> seq
> received 2d
> Jul 21 15:59:26 test openais[11845]: [TOTEM] entering COMMIT state.
> Jul 21 15:59:26 test openais[11845]: [TOTEM] entering RECOVERY state.
> Jul 21 15:59:26 test openais[11845]: [TOTEM] position [0] member
> 206.122.131.141:
> Jul 21 15:59:26 test openais[11845]: [TOTEM] previous ring seq 116
> rep
> 127.0.0.1
> Jul 21 15:59:26 test openais[11845]: [TOTEM] aru 2d high delivered 2d
> received flag 0
> Jul 21 15:59:26 test openais[11845]: [TOTEM] Did not need to
> originate any
> messages in recovery.
> Jul 21 15:59:26 test openais[11845]: [TOTEM] Storing new sequence id
> for
> ring 78
> Jul 21 15:59:26 test openais[11845]: [TOTEM] Sending initial ORF
> token
> Jul 21 15:59:26 test openais[11845]: [CLM  ] CLM CONFIGURATION CHANGE
> Jul 21 15:59:26 test openais[11845]: [CLM  ] New Configuration:
> Jul 21 15:59:26 test openais[11845]: [CLM  ] #011r(0)
> ip(206.122.131.141)
> Jul 21 15:59:26 test openais[11845]: [CLM  ] Members Left:
> Jul 21 15:59:26 test openais[11845]: [CLM  ] Members Joined:
> Jul 21 15:59:26 test openais[11845]: [SYNC ] This node is within the
> primary
> component and will provide service.
> Jul 21 15:59:26 test openais[11845]: [CLM  ] CLM CONFIGURATION CHANGE
> Jul 21 15:59:26 test openais[11845]: [CLM  ] New Configuration:
> Jul 21 15:59:26 test openais[11845]: [CLM  ] #011r(0)
> ip(206.122.131.141)
> Jul 21 15:59:26 test openais[11845]: [CLM  ] Members Left:
> Jul 21 15:59:26 test openais[11845]: [CLM  ] Members Joined:
> Jul 21 15:59:26 test openais[11845]: [SYNC ] This node is within the
> primary
> component and will provide service.
> Jul 21 15:59:26 test openais[11845]: [TOTEM] entering OPERATIONAL
> state.
> Jul 21 15:59:26 test openais[11845]: [CLM  ] got nodejoin message
> 206.122.131.141
> Jul 21 15:59:26 test openais[11845]: [CPG  ] got joinlist message
> from node
> -1920763186
> Jul 21 15:59:27 test kernel: igb: eth0 NIC Link is Up 1000 Mbps Full
> Duplex,
> Flow Control: RX
> Jul 21 15:59:27 test kernel: ADDRCONF(NETDEV_CHANGE): eth0: link
> becomes
> ready
> 
> 
> 
> 
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/network-interface-down-and-up-cluster-start-failed-tp7580087.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message