cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CASSANDRA-5332) 1.2.2 Nodetool reports 1.1.6 nodes as down
Date Mon, 11 Mar 2013 18:09:14 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Brandon Williams resolved CASSANDRA-5332.
-----------------------------------------

    Resolution: Duplicate

See CASSANDRA-5102.  The minimum minor revision you can upgrade from is 1.1.7
                
> 1.2.2 Nodetool reports 1.1.6 nodes as down
> ------------------------------------------
>
>                 Key: CASSANDRA-5332
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5332
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Sun Java 6_u39
> Ubuntu 10.04.1
> Cassandra 1.2.2 and 1.1.6
>            Reporter: Arya Goudarzi
>
> During exercise of a rolling upgrade from 1.1.6 to 1.2.2, I upgraded one node from 1.1.6
to 1.2.2 today. It appears that node running 1.2.2 reports nodes in 1.1.6 as Down in nodetool.
This means that the nodes running 1.1.6 see all other nodes including 1.2.2 as Up. Here is
the ring and gossip from nodes with 1.1.6 for example. "231.121" is the upgraded node:
> Address         DC          Rack        Status State   Load            Effective-Ownership
Token
>                                                                                     
      141784319550391026443072753098378663700
> XX.180.36    us-east     1b          Up     Normal  49.47 GB        25.00%          
   1808575600
> XX.231.121  us-east     1c          Up     Normal  47.08 GB        25.00%           
  7089215977519551322153637656637080005
> XX.177.177  us-east     1d          Up     Normal  33.64 GB        25.00%           
  14178431955039102644307275311465584410
> XX.7.148    us-east     1b          Up     Normal  41.27 GB        25.00%           
  42535295865117307932921825930779602030
> XX.20.9     us-east     1c          Up     Normal  38.51 GB        25.00%           
  49624511842636859255075463585608106435
> XX.86.255    us-east     1d          Up     Normal  34.78 GB        25.00%          
   56713727820156410577229101240436610840
> XX.63.230    us-east     1b          Up     Normal  38.11 GB        25.00%          
   85070591730234615865843651859750628460
> XX.163.36   us-east     1c          Up     Normal  44.25 GB        25.00%           
  92159807707754167187997289514579132865
> XX.31.234    us-east     1d          Up     Normal  44.66 GB        25.00%          
   99249023685273718510150927169407637270
> XX.132.169   us-east     1b          Up     Normal  44.2 GB         25.00%          
   127605887595351923798765477788721654890
> XX.71.63     us-east     1c          Up     Normal  38.74 GB        25.00%          
   134695103572871475120919115443550159295
> XX.197.209  us-east     1d          Up     Normal  41.5 GB         25.00%           
  141784319550391026443072753098378663700
> /XX.71.63
>   RACK:1c
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.1598705272E10
>   DC:us-east
>   INTERNAL_IP:XX.194.92
>   STATUS:NORMAL,134695103572871475120919115443550159295
>   RPC_ADDRESS:XX.194.92
>   RELEASE_VERSION:1.1.6
> /XX.86.255
>   RACK:1d
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:3.734334162E10
>   DC:us-east
>   INTERNAL_IP:XX.6.195
>   STATUS:NORMAL,56713727820156410577229101240436610840
>   RPC_ADDRESS:XX.6.195
>   RELEASE_VERSION:1.1.6
> /XX.7.148
>   RACK:1b
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.4316975808E10
>   DC:us-east
>   INTERNAL_IP:XX.47.250
>   STATUS:NORMAL,42535295865117307932921825930779602030
>   RPC_ADDRESS:XX.47.250
>   RELEASE_VERSION:1.1.6
> /XX.63.230
>   RACK:1b
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.0918593305E10
>   DC:us-east
>   INTERNAL_IP:XX.89.127
>   STATUS:NORMAL,85070591730234615865843651859750628460
>   RPC_ADDRESS:XX.89.127
>   RELEASE_VERSION:1.1.6
> /XX.132.169
>   RACK:1b
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.745883458E10
>   DC:us-east
>   INTERNAL_IP:XX.94.161
>   STATUS:NORMAL,127605887595351923798765477788721654890
>   RPC_ADDRESS:XX.94.161
>   RELEASE_VERSION:1.1.6
> /XX.180.36
>   RACK:1b
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:5.311963027E10
>   DC:us-east
>   INTERNAL_IP:XX.123.112
>   STATUS:NORMAL,1808575600
>   RPC_ADDRESS:XX.123.112
>   RELEASE_VERSION:1.1.6
> /XX.163.36
>   RACK:1c
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.7516755022E10
>   DC:us-east
>   INTERNAL_IP:XX.163.180
>   STATUS:NORMAL,92159807707754167187997289514579132865
>   RPC_ADDRESS:XX.163.180
>   RELEASE_VERSION:1.1.6
> /XX.31.234
>   RACK:1d
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.7954372912E10
>   DC:us-east
>   INTERNAL_IP:XX.192.159
>   STATUS:NORMAL,99249023685273718510150927169407637270
>   RPC_ADDRESS:XX.192.159
>   RELEASE_VERSION:1.1.6
> /XX.197.209
>   RACK:1d
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.4558968005E10
>   DC:us-east
>   INTERNAL_IP:XX.66.205
>   STATUS:NORMAL,141784319550391026443072753098378663700
>   RPC_ADDRESS:XX.66.205
>   RELEASE_VERSION:1.1.6
> /XX.177.177
>   RACK:1d
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:3.6115572697E10
>   DC:us-east
>   INTERNAL_IP:XX.65.57
>   STATUS:NORMAL,14178431955039102644307275311465584410
>   RPC_ADDRESS:XX.65.57
>   RELEASE_VERSION:1.1.6
> /XX.20.9
>   RACK:1c
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.1352503882E10
>   DC:us-east
>   INTERNAL_IP:XX.33.229
>   STATUS:NORMAL,49624511842636859255075463585608106435
>   RPC_ADDRESS:XX.33.229
>   RELEASE_VERSION:1.1.6
> /XX.231.121
>   RACK:1c
>   SCHEMA:09487aa5-3380-33ab-b9a5-bcc8476066b0
>   X4:9c765678-d058-4d85-a588-638ce10ff984
>   X3:7
>   DC:us-east
>   INTERNAL_IP:XX.223.241
>   RPC_ADDRESS:XX.223.241
>   RELEASE_VERSION:1.2.2
> Now the nodetool on the 1.2.2 node shows all nodes as Down but itself. Gossipinfo looks
good though:
> Datacenter: us-east
> ==========
> Replicas: 3
> Address         Rack        Status State   Load            Owns                Token
>                                                                                56713727820156410577229101240436610840
> XX.132.169   1b          Down   Normal  44.2 GB         25.00%              127605887595351923798765477788721654890
> XX.7.148    1b          Down   Normal  41.27 GB        25.00%              42535295865117307932921825930779602030
> XX.180.36    1b          Down   Normal  49.47 GB        25.00%              1808575600
> XX.63.230    1b          Down   Normal  38.11 GB        25.00%              85070591730234615865843651859750628460
> XX.231.121  1c          Up     Normal  47.25 GB        25.00%              7089215977519551322153637656637080005
> XX.71.63     1c          Down   Normal  38.74 GB        25.00%              134695103572871475120919115443550159295
> XX.177.177  1d          Down   Normal  33.64 GB        25.00%              14178431955039102644307275311465584410
> XX.31.234    1d          Down   Normal  44.66 GB        25.00%              99249023685273718510150927169407637270
> XX.20.9     1c          Down   Normal  38.51 GB        25.00%              49624511842636859255075463585608106435
> XX.163.36   1c          Down   Normal  44.25 GB        25.00%              92159807707754167187997289514579132865
> XX.197.209  1d          Down   Normal  41.5 GB         25.00%              141784319550391026443072753098378663700
> XX.86.255    1d          Down   Normal  34.78 GB        25.00%              56713727820156410577229101240436610840
> /XX.71.63
>   RACK:1c
>   RPC_ADDRESS:XX.194.92
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.194.92
>   STATUS:NORMAL,134695103572871475120919115443550159295
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.1598705272E10
> /XX.86.255
>   RACK:1d
>   RPC_ADDRESS:XX.6.195
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.6.195
>   STATUS:NORMAL,56713727820156410577229101240436610840
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:3.7343205002E10
> /XX.7.148
>   RACK:1b
>   RPC_ADDRESS:XX.47.250
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.47.250
>   STATUS:NORMAL,42535295865117307932921825930779602030
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.4316975808E10
> /XX.63.230
>   RACK:1b
>   RPC_ADDRESS:XX.89.127
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.89.127
>   STATUS:NORMAL,85070591730234615865843651859750628460
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.0918456687E10
> /XX.132.169
>   RACK:1b
>   RPC_ADDRESS:XX.94.161
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.94.161
>   STATUS:NORMAL,127605887595351923798765477788721654890
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.745883458E10
> /XX.180.36
>   RACK:1b
>   RPC_ADDRESS:XX.123.112
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.123.112
>   STATUS:NORMAL,1808575600
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:5.311963027E10
> /XX.163.36
>   RACK:1c
>   RPC_ADDRESS:XX.163.180
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.163.180
>   STATUS:NORMAL,92159807707754167187997289514579132865
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.7516755022E10
> /XX.31.234
>   RACK:1d
>   RPC_ADDRESS:XX.192.159
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.192.159
>   STATUS:NORMAL,99249023685273718510150927169407637270
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.7954372912E10
> /XX.197.209
>   RACK:1d
>   RPC_ADDRESS:XX.66.205
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.66.205
>   STATUS:NORMAL,141784319550391026443072753098378663700
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.4559013211E10
> /XX.177.177
>   RACK:1d
>   RPC_ADDRESS:XX.65.57
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.65.57
>   STATUS:NORMAL,14178431955039102644307275311465584410
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:3.6115572697E10
> /XX.20.9
>   RACK:1c
>   RPC_ADDRESS:XX.33.229
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.33.229
>   STATUS:NORMAL,49624511842636859255075463585608106435
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.1352367264E10
> /XX.231.121
>   HOST_ID:9c765678-d058-4d85-a588-638ce10ff984
>   RACK:1c
>   RPC_ADDRESS:XX.223.241
>   RELEASE_VERSION:1.2.2
>   INTERNAL_IP:XX.223.241
>   STATUS:NORMAL,7089215977519551322153637656637080005
>   NET_VERSION:7
>   SCHEMA:8b8948f5-d56f-3a96-8005-b9452e42cd67
>   SEVERITY:0.0
>   DC:us-east
>   LOAD:5.0710624207E10
> Is this just a display bug in nodetool or this upgraded node really sees the other ones
as dead?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message