Return-Path: X-Original-To: apmail-qpid-users-archive@www.apache.org Delivered-To: apmail-qpid-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AAD54D9E8 for ; Wed, 25 Jul 2012 11:04:36 +0000 (UTC) Received: (qmail 71294 invoked by uid 500); 25 Jul 2012 11:04:36 -0000 Delivered-To: apmail-qpid-users-archive@qpid.apache.org Received: (qmail 71139 invoked by uid 500); 25 Jul 2012 11:04:35 -0000 Mailing-List: contact users-help@qpid.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@qpid.apache.org Delivered-To: mailing list users@qpid.apache.org Received: (qmail 71119 invoked by uid 99); 25 Jul 2012 11:04:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Jul 2012 11:04:35 +0000 X-ASF-Spam-Status: No, hits=-3.7 required=5.0 tests=RCVD_IN_DNSWL_HI,SPF_HELO_PASS,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of pmoravec@redhat.com designates 209.132.183.25 as permitted sender) Received: from [209.132.183.25] (HELO mx4-phx2.redhat.com) (209.132.183.25) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Jul 2012 11:04:30 +0000 Received: from mail02.corp.redhat.com (zmail02.collab.prod.int.phx2.redhat.com [10.5.5.42]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q6PB49UQ023590 for ; Wed, 25 Jul 2012 07:04:09 -0400 Date: Wed, 25 Jul 2012 07:04:09 -0400 (EDT) From: Pavel Moravec To: users@qpid.apache.org Subject: Re: network interface down and up, cluster start failed Message-ID: <77760be8-71b0-42c8-bf1c-e5c75ac5361e@zmail02.collab.prod.int.phx2.redhat.com> In-Reply-To: <1343117062166-7580087.post@n2.nabble.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Originating-IP: [10.34.1.224] X-Mailer: Zimbra 7.1.2_GA_3268 (ZimbraWebClient - FF3.0 (Linux)/7.1.2_GA_3268) X-Virus-Checked: Checked by ClamAV on apache.org Hi, what is the purpose of shutting down network interface for few minutes? Jus= t a test simulating a network failure? If so, then I recommend using either= of these methods that simulate so more closely (and no issues should occur= ): 1. Pull the network cable that is used for cluster communication on one = of the cluster nodes. 2. Bring down the corresponding port on the switch. 3. Use iptables to drop both incoming and outgoing traffic for the IP as= sociated with heartbeat traffic for the cluster node. There is a script on = the the Github.com repository for corosync that be used: https://github.com= /corosync/corosync/blob/master/cts/agents/net_breaker.sh $ ~/bin/net_breaker BreakCommCmd 192.168.1.101 Kind regards, Pavel ----- Original Message ----- > From: "sun198507" > To: users@qpid.apache.org > Sent: Tuesday, July 24, 2012 10:04:22 AM > Subject: network interface down and up, cluster start failed > > Dear All! > I am testing cluster of cpp qpid. Testing cluster only include a > node.Qpid > is 0.14 release,openais is 0.8.0.3.When network is ok, everything is > ok.But > When I down the network interface , wait for a few minutes and up the > network interface, the cluster node start failed for "critical > Unexpected > error: Daemon startup failed: Cannot join CPG group msgplat: try > again (6)". > Restart openais, Qpid start success. More detailed failed messages as > follows. Anybody can tell me why the case happen or give some ideas? > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D > Jul 21 13:53:29 test kernel: ADDRCONF(NETDEV_UP): eth0: link is not > ready > Jul 21 13:53:29 test openais[2222]: [TOTEM] Receive multicast socket > recv > buffer size (262142 bytes). > Jul 21 13:53:29 test openais[2222]: [TOTEM] Transmit multicast socket > send > buffer size (262142 bytes). > Jul 21 13:53:29 test openais[2222]: [TOTEM] The network interface is > down. > Jul 21 13:53:29 test openais[2222]: [TOTEM] entering GATHER state > from 15. > Jul 21 13:53:30 test openais[2222]: [TOTEM] entering GATHER state > from 0. > Jul 21 13:53:30 test openais[2222]: [TOTEM] Creating commit token > because I > am the rep. > Jul 21 13:53:30 test openais[2222]: [TOTEM] Saving state aru 8b4d06 > high seq > received 8b4d06 > Jul 21 13:53:30 test openais[2222]: [TOTEM] entering COMMIT state. > Jul 21 13:53:30 test openais[2222]: [TOTEM] entering RECOVERY state. > Jul 21 13:53:30 test openais[2222]: [TOTEM] position [0] member > 127.0.0.1: > Jul 21 13:53:30 test openais[2222]: [TOTEM] previous ring seq 100 rep > 206.122.131.141 > Jul 21 13:53:30 test openais[2222]: [TOTEM] aru 8b4d06 high delivered > 8b4d06 > received flag 0 > Jul 21 13:53:30 test openais[2222]: [TOTEM] Did not need to originate > any > messages in recovery. > Jul 21 13:53:30 test openais[2222]: [TOTEM] Storing new sequence id > for ring > 68 > Jul 21 13:53:30 test openais[2222]: [TOTEM] Sending initial ORF token > Jul 21 13:53:30 test openais[2222]: [CLM ] CLM CONFIGURATION CHANGE > Jul 21 13:53:30 test openais[2222]: [CLM ] New Configuration: > Jul 21 13:53:30 test openais[2222]: [CLM ] #011r(0) ip(127.0.0.1) > Jul 21 13:53:30 test openais[2222]: [CLM ] Members Left: > Jul 21 13:53:30 test openais[2222]: [CLM ] Members Joined: > Jul 21 13:53:30 test openais[2222]: [SYNC ] This node is within the > primary > component and will provide service. > Jul 21 13:53:30 test openais[2222]: [CLM ] CLM CONFIGURATION CHANGE > Jul 21 13:53:30 test openais[2222]: [CLM ] New Configuration: > Jul 21 13:53:30 test openais[2222]: [CLM ] #011r(0) ip(127.0.0.1) > Jul 21 13:53:30 test openais[2222]: [CLM ] Members Left: > Jul 21 13:53:30 test openais[2222]: [CLM ] Members Joined: > Jul 21 13:53:30 test openais[2222]: [SYNC ] This node is within the > primary > component and will provide service. > Jul 21 13:53:30 test openais[2222]: [TOTEM] entering OPERATIONAL > state. > Jul 21 13:53:30 test openais[2222]: [TOTEM] Message continuation > doesn't > match previous frag e: 0 - a: 146 > Jul 21 13:53:30 test openais[2222]: [TOTEM] Throwing away broken > message: > continuation 0, index 0 > Jul 21 13:53:30 test openais[2222]: [CLM ] got nodejoin message > 127.0.0.1 > Jul 21 13:53:32 test kernel: igb: eth0 NIC Link is Up 1000 Mbps Full > Duplex, > Flow Control: RX > Jul 21 13:53:32 test kernel: ADDRCONF(NETDEV_CHANGE): eth0: link > becomes > ready > Jul 21 13:53:33 test MsgBroker[2263]: 2012-07-21 13:53:33 critical > Error > delivering frames: Unknown connection: Frame[BEbe; channel=3D1; > {QueueQueryBody: queue=3D1ClueVersion; }] data 127.0.0.1:2263-124293 > read-credit=3D1 (qpid/cluster/Cluster.cpp:544) > Jul 21 13:53:33 test MsgBroker[2263]: 2012-07-21 13:53:33 notice > cluster(206.122.131.141:2263 LEFT) leaving cluster msgplat > Jul 21 13:53:33 test MsgBroker[2263]: 2012-07-21 13:53:33 notice Shut > down > Jul 21 13:53:34 test openais[2222]: [TOTEM] Receive multicast socket > recv > buffer size (262142 bytes). > Jul 21 13:53:34 test openais[2222]: [TOTEM] Transmit multicast socket > send > buffer size (262142 bytes). > Jul 21 13:53:34 test openais[2222]: [TOTEM] The network interface > [206.122.131.141] is now up. > Jul 21 13:53:34 test openais[2222]: [TOTEM] entering GATHER state > from 15. > Jul 21 13:53:35 test openais[2222]: [TOTEM] entering GATHER state > from 0. > Jul 21 13:53:35 test openais[2222]: [TOTEM] Creating commit token > because I > am the rep. > Jul 21 13:53:35 test openais[2222]: [TOTEM] Saving state aru 18a high > seq > received 18a > Jul 21 13:53:35 test openais[2222]: [TOTEM] entering COMMIT state. > Jul 21 13:53:35 test openais[2222]: [TOTEM] entering RECOVERY state. > Jul 21 13:53:35 test openais[2222]: [TOTEM] position [0] member > 206.122.131.141: > Jul 21 13:53:35 test openais[2222]: [TOTEM] previous ring seq 104 rep > 127.0.0.1 > Jul 21 13:53:35 test openais[2222]: [TOTEM] aru 18a high delivered > 18a > received flag 0 > Jul 21 13:53:35 test openais[2222]: [TOTEM] Did not need to originate > any > messages in recovery. > Jul 21 13:53:35 test openais[2222]: [TOTEM] Storing new sequence id > for ring > 6c > Jul 21 13:53:35 test openais[2222]: [TOTEM] Sending initial ORF token > Jul 21 13:53:35 test openais[2222]: [CLM ] CLM CONFIGURATION CHANGE > Jul 21 13:53:35 test openais[2222]: [CLM ] New Configuration: > Jul 21 13:53:35 test openais[2222]: [CLM ] #011r(0) > ip(206.122.131.141) > Jul 21 13:53:35 test openais[2222]: [CLM ] Members Left: > Jul 21 13:53:35 test openais[2222]: [CLM ] Members Joined: > Jul 21 13:53:35 test openais[2222]: [SYNC ] This node is within the > primary > component and will provide service. > Jul 21 13:53:35 test openais[2222]: [CLM ] CLM CONFIGURATION CHANGE > Jul 21 13:53:35 test openais[2222]: [CLM ] New Configuration: > Jul 21 13:53:35 test openais[2222]: [CLM ] #011r(0) > ip(206.122.131.141) > Jul 21 13:53:35 test openais[2222]: [CLM ] Members Left: > Jul 21 13:53:35 test openais[2222]: [CLM ] Members Joined: > Jul 21 13:53:35 test openais[2222]: [SYNC ] This node is within the > primary > component and will provide service. > Jul 21 13:53:35 test openais[2222]: [TOTEM] entering OPERATIONAL > state. > Jul 21 13:53:48 test MsgBroker[29877]: 2012-07-21 13:53:48 notice > Starting > watchdog process with interval of 60 seconds > Jul 21 13:53:48 test MsgBroker[29877]: 2012-07-21 13:53:48 notice > Initializing CPG > Jul 21 13:53:48 test MsgBroker[29877]: 2012-07-21 13:53:48 critical > Unexpected error: Cannot join CPG group msgplat: try again (6) > Jul 21 13:53:48 test MsgBroker[29876]: 2012-07-21 13:53:48 critical > Unexpected error: Daemon startup failed: Cannot join CPG group > msgplat: try > again (6) > Jul 21 13:54:09 test MsgBroker[29932]: 2012-07-21 13:54:09 notice > Starting > watchdog process with interval of 60 seconds > Jul 21 13:54:09 test MsgBroker[29932]: 2012-07-21 13:54:09 notice > Initializing CPG > Jul 21 13:54:09 test MsgBroker[29932]: 2012-07-21 13:54:09 critical > Unexpected error: Cannot join CPG group msgplat: try again (6) > Jul 21 13:54:09 test MsgBroker[29931]: 2012-07-21 13:54:09 critical > Unexpected error: Daemon startup failed: Cannot join CPG group > msgplat: try > again (6) > Jul 21 13:54:29 test MsgBroker[29989]: 2012-07-21 13:54:29 notice > Starting > watchdog process with interval of 60 seconds > Jul 21 13:54:29 test MsgBroker[29989]: 2012-07-21 13:54:29 notice > Initializing CPG > Jul 21 13:54:29 test MsgBroker[29989]: 2012-07-21 13:54:29 critical > Unexpected error: Cannot join CPG group msgplat: try again (6) > Jul 21 13:54:29 test MsgBroker[29988]: 2012-07-21 13:54:29 critical > Unexpected error: Daemon startup failed: Cannot join CPG group > msgplat: try > again (6) > Jul 21 13:54:49 test MsgBroker[30044]: 2012-07-21 13:54:49 notice > Starting > watchdog process with interval of 60 seconds > Jul 21 13:54:49 test MsgBroker[30044]: 2012-07-21 13:54:49 notice > Initializing CPG > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > At the same test with network interface down and up, Qpid Broker hang > for > critical Unexpected error: Timed out waiting for daemon (If store > recovery > is in progress, use longer wait time)=EF=BC=8C not accept any connection = from > client > until restart aisexec and Qpid. Detailed failed messages as follows. > > > Jul 21 15:16:28 test openais[11845]: [MAIN ] AIS Executive Service > RELEASE > 'subrev 1358 version 0.80.3' > Jul 21 15:16:28 test openais[11845]: [MAIN ] Copyright (C) 2002-2006 > MontaVista Software, Inc and contributors. > Jul 21 15:16:28 test openais[11845]: [MAIN ] Copyright (C) 2006 Red > Hat, > Inc. > Jul 21 15:16:28 test openais[11845]: [MAIN ] AIS Executive Service: > started > and ready to provide service. > Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component > openais_cpg > loaded. > Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service > handler > 'openais cluster closed process group service v1.01' > Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component > openais_cfg > loaded. > Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service > handler > 'openais configuration service' > Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component > openais_msg > loaded. > Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service > handler > 'openais message service B.01.01' > Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component > openais_lck > loaded. > Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service > handler > 'openais distributed locking service B.01.01' > Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component > openais_evt > loaded. > Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service > handler > 'openais event service B.01.01' > Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component > openais_ckpt > loaded. > Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service > handler > 'openais checkpoint service B.01.01' > Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component > openais_amf > loaded. > Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service > handler > 'openais availability management framework B.01.01' > Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component > openais_clm > loaded. > Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service > handler > 'openais cluster membership service B.01.01' > Jul 21 15:16:28 test openais[11845]: [MAIN ] openais component > openais_evs > loaded. > Jul 21 15:16:28 test openais[11845]: [MAIN ] Registering service > handler > 'openais extended virtual synchrony service' > Jul 21 15:16:28 test openais[11845]: [TOTEM] Token Timeout (1000 ms) > retransmit timeout (238 ms) > Jul 21 15:16:28 test openais[11845]: [TOTEM] token hold (180 ms) > retransmits > before loss (4 retrans) > Jul 21 15:16:28 test openais[11845]: [TOTEM] join (50 ms) send_join > (0 ms) > consensus (800 ms) merge (200 ms) > Jul 21 15:16:28 test openais[11845]: [TOTEM] downcheck (1000 ms) fail > to > recv const (50 msgs) > Jul 21 15:16:28 test openais[11845]: [TOTEM] seqno unchanged const > (30 > rotations) Maximum network MTU 1500 > Jul 21 15:16:28 test openais[11845]: [TOTEM] window size per rotation > (50 > messages) maximum messages per rotation (17 messages) > Jul 21 15:16:28 test openais[11845]: [TOTEM] send threads (0 threads) > Jul 21 15:16:28 test openais[11845]: [TOTEM] RRP token expired > timeout (238 > ms) > Jul 21 15:16:28 test openais[11845]: [TOTEM] RRP token problem > counter (2000 > ms) > Jul 21 15:16:28 test openais[11845]: [TOTEM] RRP threshold (10 > problem > count) > Jul 21 15:16:28 test openais[11845]: [TOTEM] RRP mode set to none. > Jul 21 15:16:28 test openais[11845]: [TOTEM] > heartbeat_failures_allowed (0) > Jul 21 15:16:28 test openais[11845]: [TOTEM] max_network_delay (50 > ms) > Jul 21 15:16:28 test openais[11845]: [TOTEM] HeartBeat is Disabled. > To > enable set heartbeat_failures_allowed > 0 > Jul 21 15:16:28 test openais[11845]: [TOTEM] Receive multicast socket > recv > buffer size (262142 bytes). > Jul 21 15:16:28 test openais[11845]: [TOTEM] Transmit multicast > socket send > buffer size (262142 bytes). > Jul 21 15:16:28 test openais[11845]: [TOTEM] The network interface > [206.122.131.141] is now up. > Jul 21 15:16:28 test openais[11845]: [TOTEM] Created or loaded > sequence id > 108.206.122.131.141 for this ring. > Jul 21 15:16:28 test openais[11845]: [TOTEM] entering GATHER state > from 15. > Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service > handler > 'openais extended virtual synchrony service' > Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service > handler > 'openais cluster membership service B.01.01' > Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service > handler > 'openais availability management framework B.01.01' > Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service > handler > 'openais checkpoint service B.01.01' > Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service > handler > 'openais event service B.01.01' > Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service > handler > 'openais distributed locking service B.01.01' > Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service > handler > 'openais message service B.01.01' > Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service > handler > 'openais configuration service' > Jul 21 15:16:28 test openais[11845]: [SERV ] Initialising service > handler > 'openais cluster closed process group service v1.01' > Jul 21 15:16:28 test openais[11845]: [SYNC ] Not using a virtual > synchrony > filter. > Jul 21 15:16:28 test openais[11845]: [TOTEM] Creating commit token > because I > am the rep. > Jul 21 15:16:28 test openais[11845]: [TOTEM] Saving state aru 0 high > seq > received 0 > Jul 21 15:16:28 test openais[11845]: [TOTEM] entering COMMIT state. > Jul 21 15:16:28 test openais[11845]: [TOTEM] entering RECOVERY state. > Jul 21 15:16:28 test openais[11845]: [TOTEM] position [0] member > 206.122.131.141: > Jul 21 15:16:28 test openais[11845]: [TOTEM] previous ring seq 108 > rep > 206.122.131.141 > Jul 21 15:16:28 test openais[11845]: [TOTEM] aru 0 high delivered 0 > received > flag 0 > Jul 21 15:16:28 test openais[11845]: [TOTEM] Did not need to > originate any > messages in recovery. > Jul 21 15:16:28 test openais[11845]: [TOTEM] Storing new sequence id > for > ring 70 > Jul 21 15:16:28 test openais[11845]: [TOTEM] Sending initial ORF > token > Jul 21 15:16:28 test openais[11845]: [CLM ] CLM CONFIGURATION CHANGE > Jul 21 15:16:28 test openais[11845]: [CLM ] New Configuration: > Jul 21 15:16:28 test openais[11845]: [CLM ] Members Left: > Jul 21 15:16:28 test openais[11845]: [CLM ] Members Joined: > Jul 21 15:16:28 test openais[11845]: [SYNC ] This node is within the > primary > component and will provide service. > Jul 21 15:16:28 test openais[11845]: [CLM ] CLM CONFIGURATION CHANGE > Jul 21 15:16:28 test openais[11845]: [CLM ] New Configuration: > Jul 21 15:16:28 test openais[11845]: [CLM ] #011r(0) > ip(206.122.131.141) > Jul 21 15:16:28 test openais[11845]: [CLM ] Members Left: > Jul 21 15:16:28 test openais[11845]: [CLM ] Members Joined: > Jul 21 15:16:28 test openais[11845]: [CLM ] #011r(0) > ip(206.122.131.141) > Jul 21 15:16:28 test openais[11845]: [SYNC ] This node is within the > primary > component and will provide service. > Jul 21 15:16:28 test openais[11845]: [TOTEM] entering OPERATIONAL > state. > Jul 21 15:16:28 test openais[11845]: [CLM ] got nodejoin message > 206.122.131.141 > Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice > Starting > watchdog process with interval of 60 seconds > Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice > Initializing CPG > Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice > cluster(206.122.131.141:11883 PRE_INIT) configuration change: > 206.122.131.141:11883 > Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice > cluster(206.122.131.141:11883 PRE_INIT) Members joined: > 206.122.131.141:11883 > Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice > SASL > disabled: No Authentication Performed > Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice > Listening > on TCP/TCP6 port 5672 > Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice > cluster(206.122.131.141:11883 INIT) cluster-uuid =3D > a5d98d15-c518-4de7-8e1a-94dea116844d > Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice > cluster(206.122.131.141:11883 READY) joined cluster msgplat > Jul 21 15:16:36 test MsgBroker[11883]: 2012-07-21 15:16:36 notice > Broker > running > Jul 21 15:16:37 test MsgBroker[11883]: 2012-07-21 15:16:37 notice > Shut down > Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice > Starting > watchdog process with interval of 60 seconds > Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice > Initializing CPG > Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice > cluster(206.122.131.141:11929 PRE_INIT) configuration change: > 206.122.131.141:11929 > Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice > cluster(206.122.131.141:11929 PRE_INIT) Members joined: > 206.122.131.141:11929 > Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice > SASL > disabled: No Authentication Performed > Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice > Listening > on TCP/TCP6 port 5672 > Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice > cluster(206.122.131.141:11929 INIT) cluster-uuid =3D > cd26ef10-d608-42ad-8e37-607f25798127 > Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice > cluster(206.122.131.141:11929 READY) joined cluster msgplat > Jul 21 15:16:49 test MsgBroker[11929]: 2012-07-21 15:16:49 notice > Broker > running > Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice > Starting > watchdog process with interval of 60 seconds > Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice > Initializing CPG > Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice > cluster(206.122.131.141:11992 PRE_INIT) configuration change: > 206.122.131.141:11992 > Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice > cluster(206.122.131.141:11992 PRE_INIT) Members joined: > 206.122.131.141:11992 > Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice > SASL > disabled: No Authentication Performed > Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice > Listening > on TCP/TCP6 port 5672 > Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice > cluster(206.122.131.141:11992 INIT) cluster-uuid =3D > a64ce087-5874-43eb-9e17-0d2aeec5e67f > Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice > cluster(206.122.131.141:11992 READY) joined cluster msgplat > Jul 21 15:17:41 test MsgBroker[11992]: 2012-07-21 15:17:41 notice > Broker > running > Jul 21 15:45:02 test openais[11845]: [TOTEM] Receive multicast socket > recv > buffer size (262142 bytes). > Jul 21 15:45:02 test openais[11845]: [TOTEM] Transmit multicast > socket send > buffer size (262142 bytes). > Jul 21 15:45:02 test openais[11845]: [TOTEM] The network interface is > down. > Jul 21 15:45:02 test openais[11845]: [TOTEM] entering GATHER state > from 15. > Jul 21 15:45:03 test openais[11845]: [TOTEM] entering GATHER state > from 0. > Jul 21 15:45:03 test openais[11845]: [TOTEM] Creating commit token > because I > am the rep. > Jul 21 15:45:03 test openais[11845]: [TOTEM] Saving state aru 31b25 > high seq > received 31b25 > Jul 21 15:45:03 test openais[11845]: [TOTEM] entering COMMIT state. > Jul 21 15:45:03 test openais[11845]: [TOTEM] entering RECOVERY state. > Jul 21 15:45:03 test openais[11845]: [TOTEM] position [0] member > 127.0.0.1: > Jul 21 15:45:03 test openais[11845]: [TOTEM] previous ring seq 112 > rep > 206.122.131.141 > Jul 21 15:45:03 test openais[11845]: [TOTEM] aru 31b25 high delivered > 31b25 > received flag 0 > Jul 21 15:45:03 test openais[11845]: [TOTEM] Did not need to > originate any > messages in recovery. > Jul 21 15:45:03 test openais[11845]: [TOTEM] Storing new sequence id > for > ring 74 > Jul 21 15:45:03 test openais[11845]: [TOTEM] Sending initial ORF > token > Jul 21 15:45:03 test openais[11845]: [CLM ] CLM CONFIGURATION CHANGE > Jul 21 15:45:03 test openais[11845]: [CLM ] New Configuration: > Jul 21 15:45:03 test openais[11845]: [CLM ] #011r(0) ip(127.0.0.1) > Jul 21 15:45:03 test openais[11845]: [CLM ] Members Left: > Jul 21 15:45:03 test openais[11845]: [CLM ] Members Joined: > Jul 21 15:45:03 test openais[11845]: [SYNC ] This node is within the > primary > component and will provide service. > Jul 21 15:45:03 test openais[11845]: [CLM ] CLM CONFIGURATION CHANGE > Jul 21 15:45:03 test openais[11845]: [CLM ] New Configuration: > Jul 21 15:45:03 test openais[11845]: [CLM ] #011r(0) ip(127.0.0.1) > Jul 21 15:45:03 test openais[11845]: [CLM ] Members Left: > Jul 21 15:45:03 test openais[11845]: [CLM ] Members Joined: > Jul 21 15:45:03 test openais[11845]: [SYNC ] This node is within the > primary > component and will provide service. > Jul 21 15:45:03 test openais[11845]: [TOTEM] entering OPERATIONAL > state. > Jul 21 15:45:03 test openais[11845]: [CLM ] got nodejoin message > 127.0.0.1 > Jul 21 15:45:03 test MsgBroker[11992]: 2012-07-21 15:45:03 critical > Error > delivering frames: Unknown connection: Frame[BEbe; channel=3D1; > {SessionDetachBody: name=3De8b38236-934b-4c2a-99c7-73bc09d91f67; }] > data > 127.0.0.1:11992-2488 read-credit=3D1 (qpid/cluster/Cluster.cpp:544) > Jul 21 15:45:03 test MsgBroker[11992]: 2012-07-21 15:45:03 notice > cluster(206.122.131.141:11992 LEFT) leaving cluster msgplat > Jul 21 15:45:03 test MsgBroker[11992]: 2012-07-21 15:45:03 notice > Shut down > Jul 21 15:45:04 test MsgBroker[15998]: 2012-07-21 15:45:04 notice > Starting > watchdog process with interval of 60 seconds > Jul 21 15:45:04 test MsgBroker[15998]: 2012-07-21 15:45:04 notice > Initializing CPG > Jul 21 15:55:04 test MsgBroker[15997]: 2012-07-21 15:55:04 critical > Unexpected error: Timed out waiting for daemon (If store recovery is > in > progress, use longer wait time) > Jul 21 15:59:25 test kernel: ADDRCONF(NETDEV_UP): eth0: link is not > ready > Jul 21 15:59:25 test openais[11845]: [TOTEM] Receive multicast socket > recv > buffer size (262142 bytes). > Jul 21 15:59:25 test openais[11845]: [TOTEM] Transmit multicast > socket send > buffer size (262142 bytes). > Jul 21 15:59:25 test openais[11845]: [TOTEM] The network interface > [206.122.131.141] is now up. > Jul 21 15:59:25 test openais[11845]: [TOTEM] entering GATHER state > from 15. > Jul 21 15:59:26 test openais[11845]: [TOTEM] entering GATHER state > from 0. > Jul 21 15:59:26 test openais[11845]: [TOTEM] Creating commit token > because I > am the rep. > Jul 21 15:59:26 test openais[11845]: [TOTEM] Saving state aru 2d high > seq > received 2d > Jul 21 15:59:26 test openais[11845]: [TOTEM] entering COMMIT state. > Jul 21 15:59:26 test openais[11845]: [TOTEM] entering RECOVERY state. > Jul 21 15:59:26 test openais[11845]: [TOTEM] position [0] member > 206.122.131.141: > Jul 21 15:59:26 test openais[11845]: [TOTEM] previous ring seq 116 > rep > 127.0.0.1 > Jul 21 15:59:26 test openais[11845]: [TOTEM] aru 2d high delivered 2d > received flag 0 > Jul 21 15:59:26 test openais[11845]: [TOTEM] Did not need to > originate any > messages in recovery. > Jul 21 15:59:26 test openais[11845]: [TOTEM] Storing new sequence id > for > ring 78 > Jul 21 15:59:26 test openais[11845]: [TOTEM] Sending initial ORF > token > Jul 21 15:59:26 test openais[11845]: [CLM ] CLM CONFIGURATION CHANGE > Jul 21 15:59:26 test openais[11845]: [CLM ] New Configuration: > Jul 21 15:59:26 test openais[11845]: [CLM ] #011r(0) > ip(206.122.131.141) > Jul 21 15:59:26 test openais[11845]: [CLM ] Members Left: > Jul 21 15:59:26 test openais[11845]: [CLM ] Members Joined: > Jul 21 15:59:26 test openais[11845]: [SYNC ] This node is within the > primary > component and will provide service. > Jul 21 15:59:26 test openais[11845]: [CLM ] CLM CONFIGURATION CHANGE > Jul 21 15:59:26 test openais[11845]: [CLM ] New Configuration: > Jul 21 15:59:26 test openais[11845]: [CLM ] #011r(0) > ip(206.122.131.141) > Jul 21 15:59:26 test openais[11845]: [CLM ] Members Left: > Jul 21 15:59:26 test openais[11845]: [CLM ] Members Joined: > Jul 21 15:59:26 test openais[11845]: [SYNC ] This node is within the > primary > component and will provide service. > Jul 21 15:59:26 test openais[11845]: [TOTEM] entering OPERATIONAL > state. > Jul 21 15:59:26 test openais[11845]: [CLM ] got nodejoin message > 206.122.131.141 > Jul 21 15:59:26 test openais[11845]: [CPG ] got joinlist message > from node > -1920763186 > Jul 21 15:59:27 test kernel: igb: eth0 NIC Link is Up 1000 Mbps Full > Duplex, > Flow Control: RX > Jul 21 15:59:27 test kernel: ADDRCONF(NETDEV_CHANGE): eth0: link > becomes > ready > > > > > -- > View this message in context: > http://qpid.2158936.n2.nabble.com/network-interface-down-and-up-cluster-s= tart-failed-tp7580087.html > Sent from the Apache Qpid users mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org > For additional commands, e-mail: users-help@qpid.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional commands, e-mail: users-help@qpid.apache.org