Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 22024E705 for ; Mon, 11 Mar 2013 18:09:16 +0000 (UTC) Received: (qmail 24830 invoked by uid 500); 11 Mar 2013 18:09:14 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 24797 invoked by uid 500); 11 Mar 2013 18:09:14 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 24754 invoked by uid 99); 11 Mar 2013 18:09:14 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Mar 2013 18:09:14 +0000 Date: Mon, 11 Mar 2013 18:09:14 +0000 (UTC) From: "Brandon Williams (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (CASSANDRA-5332) 1.2.2 Nodetool reports 1.1.6 nodes as down MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams resolved CASSANDRA-5332. ----------------------------------------- Resolution: Duplicate See CASSANDRA-5102. The minimum minor revision you can upgrade from is 1.1.7 > 1.2.2 Nodetool reports 1.1.6 nodes as down > ------------------------------------------ > > Key: CASSANDRA-5332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5332 > Project: Cassandra > Issue Type: Bug > Environment: Sun Java 6_u39 > Ubuntu 10.04.1 > Cassandra 1.2.2 and 1.1.6 > Reporter: Arya Goudarzi > > During exercise of a rolling upgrade from 1.1.6 to 1.2.2, I upgraded one node from 1.1.6 to 1.2.2 today. It appears that node running 1.2.2 reports nodes in 1.1.6 as Down in nodetool. This means that the nodes running 1.1.6 see all other nodes including 1.2.2 as Up. Here is the ring and gossip from nodes with 1.1.6 for example. "231.121" is the upgraded node: > Address DC Rack Status State Load Effective-Ownership Token > 141784319550391026443072753098378663700 > XX.180.36 us-east 1b Up Normal 49.47 GB 25.00% 1808575600 > XX.231.121 us-east 1c Up Normal 47.08 GB 25.00% 7089215977519551322153637656637080005 > XX.177.177 us-east 1d Up Normal 33.64 GB 25.00% 14178431955039102644307275311465584410 > XX.7.148 us-east 1b Up Normal 41.27 GB 25.00% 42535295865117307932921825930779602030 > XX.20.9 us-east 1c Up Normal 38.51 GB 25.00% 49624511842636859255075463585608106435 > XX.86.255 us-east 1d Up Normal 34.78 GB 25.00% 56713727820156410577229101240436610840 > XX.63.230 us-east 1b Up Normal 38.11 GB 25.00% 85070591730234615865843651859750628460 > XX.163.36 us-east 1c Up Normal 44.25 GB 25.00% 92159807707754167187997289514579132865 > XX.31.234 us-east 1d Up Normal 44.66 GB 25.00% 99249023685273718510150927169407637270 > XX.132.169 us-east 1b Up Normal 44.2 GB 25.00% 127605887595351923798765477788721654890 > XX.71.63 us-east 1c Up Normal 38.74 GB 25.00% 134695103572871475120919115443550159295 > XX.197.209 us-east 1d Up Normal 41.5 GB 25.00% 141784319550391026443072753098378663700 > /XX.71.63 > RACK:1c > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.1598705272E10 > DC:us-east > INTERNAL_IP:XX.194.92 > STATUS:NORMAL,134695103572871475120919115443550159295 > RPC_ADDRESS:XX.194.92 > RELEASE_VERSION:1.1.6 > /XX.86.255 > RACK:1d > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:3.734334162E10 > DC:us-east > INTERNAL_IP:XX.6.195 > STATUS:NORMAL,56713727820156410577229101240436610840 > RPC_ADDRESS:XX.6.195 > RELEASE_VERSION:1.1.6 > /XX.7.148 > RACK:1b > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.4316975808E10 > DC:us-east > INTERNAL_IP:XX.47.250 > STATUS:NORMAL,42535295865117307932921825930779602030 > RPC_ADDRESS:XX.47.250 > RELEASE_VERSION:1.1.6 > /XX.63.230 > RACK:1b > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.0918593305E10 > DC:us-east > INTERNAL_IP:XX.89.127 > STATUS:NORMAL,85070591730234615865843651859750628460 > RPC_ADDRESS:XX.89.127 > RELEASE_VERSION:1.1.6 > /XX.132.169 > RACK:1b > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.745883458E10 > DC:us-east > INTERNAL_IP:XX.94.161 > STATUS:NORMAL,127605887595351923798765477788721654890 > RPC_ADDRESS:XX.94.161 > RELEASE_VERSION:1.1.6 > /XX.180.36 > RACK:1b > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:5.311963027E10 > DC:us-east > INTERNAL_IP:XX.123.112 > STATUS:NORMAL,1808575600 > RPC_ADDRESS:XX.123.112 > RELEASE_VERSION:1.1.6 > /XX.163.36 > RACK:1c > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.7516755022E10 > DC:us-east > INTERNAL_IP:XX.163.180 > STATUS:NORMAL,92159807707754167187997289514579132865 > RPC_ADDRESS:XX.163.180 > RELEASE_VERSION:1.1.6 > /XX.31.234 > RACK:1d > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.7954372912E10 > DC:us-east > INTERNAL_IP:XX.192.159 > STATUS:NORMAL,99249023685273718510150927169407637270 > RPC_ADDRESS:XX.192.159 > RELEASE_VERSION:1.1.6 > /XX.197.209 > RACK:1d > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.4558968005E10 > DC:us-east > INTERNAL_IP:XX.66.205 > STATUS:NORMAL,141784319550391026443072753098378663700 > RPC_ADDRESS:XX.66.205 > RELEASE_VERSION:1.1.6 > /XX.177.177 > RACK:1d > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:3.6115572697E10 > DC:us-east > INTERNAL_IP:XX.65.57 > STATUS:NORMAL,14178431955039102644307275311465584410 > RPC_ADDRESS:XX.65.57 > RELEASE_VERSION:1.1.6 > /XX.20.9 > RACK:1c > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > LOAD:4.1352503882E10 > DC:us-east > INTERNAL_IP:XX.33.229 > STATUS:NORMAL,49624511842636859255075463585608106435 > RPC_ADDRESS:XX.33.229 > RELEASE_VERSION:1.1.6 > /XX.231.121 > RACK:1c > SCHEMA:09487aa5-3380-33ab-b9a5-bcc8476066b0 > X4:9c765678-d058-4d85-a588-638ce10ff984 > X3:7 > DC:us-east > INTERNAL_IP:XX.223.241 > RPC_ADDRESS:XX.223.241 > RELEASE_VERSION:1.2.2 > Now the nodetool on the 1.2.2 node shows all nodes as Down but itself. Gossipinfo looks good though: > Datacenter: us-east > ========== > Replicas: 3 > Address Rack Status State Load Owns Token > 56713727820156410577229101240436610840 > XX.132.169 1b Down Normal 44.2 GB 25.00% 127605887595351923798765477788721654890 > XX.7.148 1b Down Normal 41.27 GB 25.00% 42535295865117307932921825930779602030 > XX.180.36 1b Down Normal 49.47 GB 25.00% 1808575600 > XX.63.230 1b Down Normal 38.11 GB 25.00% 85070591730234615865843651859750628460 > XX.231.121 1c Up Normal 47.25 GB 25.00% 7089215977519551322153637656637080005 > XX.71.63 1c Down Normal 38.74 GB 25.00% 134695103572871475120919115443550159295 > XX.177.177 1d Down Normal 33.64 GB 25.00% 14178431955039102644307275311465584410 > XX.31.234 1d Down Normal 44.66 GB 25.00% 99249023685273718510150927169407637270 > XX.20.9 1c Down Normal 38.51 GB 25.00% 49624511842636859255075463585608106435 > XX.163.36 1c Down Normal 44.25 GB 25.00% 92159807707754167187997289514579132865 > XX.197.209 1d Down Normal 41.5 GB 25.00% 141784319550391026443072753098378663700 > XX.86.255 1d Down Normal 34.78 GB 25.00% 56713727820156410577229101240436610840 > /XX.71.63 > RACK:1c > RPC_ADDRESS:XX.194.92 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.194.92 > STATUS:NORMAL,134695103572871475120919115443550159295 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.1598705272E10 > /XX.86.255 > RACK:1d > RPC_ADDRESS:XX.6.195 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.6.195 > STATUS:NORMAL,56713727820156410577229101240436610840 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:3.7343205002E10 > /XX.7.148 > RACK:1b > RPC_ADDRESS:XX.47.250 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.47.250 > STATUS:NORMAL,42535295865117307932921825930779602030 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.4316975808E10 > /XX.63.230 > RACK:1b > RPC_ADDRESS:XX.89.127 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.89.127 > STATUS:NORMAL,85070591730234615865843651859750628460 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.0918456687E10 > /XX.132.169 > RACK:1b > RPC_ADDRESS:XX.94.161 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.94.161 > STATUS:NORMAL,127605887595351923798765477788721654890 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.745883458E10 > /XX.180.36 > RACK:1b > RPC_ADDRESS:XX.123.112 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.123.112 > STATUS:NORMAL,1808575600 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:5.311963027E10 > /XX.163.36 > RACK:1c > RPC_ADDRESS:XX.163.180 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.163.180 > STATUS:NORMAL,92159807707754167187997289514579132865 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.7516755022E10 > /XX.31.234 > RACK:1d > RPC_ADDRESS:XX.192.159 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.192.159 > STATUS:NORMAL,99249023685273718510150927169407637270 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.7954372912E10 > /XX.197.209 > RACK:1d > RPC_ADDRESS:XX.66.205 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.66.205 > STATUS:NORMAL,141784319550391026443072753098378663700 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.4559013211E10 > /XX.177.177 > RACK:1d > RPC_ADDRESS:XX.65.57 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.65.57 > STATUS:NORMAL,14178431955039102644307275311465584410 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:3.6115572697E10 > /XX.20.9 > RACK:1c > RPC_ADDRESS:XX.33.229 > RELEASE_VERSION:1.1.6 > INTERNAL_IP:XX.33.229 > STATUS:NORMAL,49624511842636859255075463585608106435 > SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575 > DC:us-east > LOAD:4.1352367264E10 > /XX.231.121 > HOST_ID:9c765678-d058-4d85-a588-638ce10ff984 > RACK:1c > RPC_ADDRESS:XX.223.241 > RELEASE_VERSION:1.2.2 > INTERNAL_IP:XX.223.241 > STATUS:NORMAL,7089215977519551322153637656637080005 > NET_VERSION:7 > SCHEMA:8b8948f5-d56f-3a96-8005-b9452e42cd67 > SEVERITY:0.0 > DC:us-east > LOAD:5.0710624207E10 > Is this just a display bug in nodetool or this upgraded node really sees the other ones as dead? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira