incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Incompatible Gossip 1.1.6 to 1.2.1 Upgrade?
Date Tue, 12 Mar 2013 05:34:58 GMT
> Is this just a display bug in nodetool or this upgraded node really sees the other ones
as dead?
Is the 1.2.2 node which is see all the others as down processing requests ? 
Is it showing the others as down in the log ? 

I'm not really sure what's happening. But you can try starting the 1.2.2 node with the 

-Dcassandra.load_ring_state=false  

parameter, append it at the bottom of the cassandra-env.sh file. It will force the node to
get the ring state from the others. 

Cheers
 
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 8/03/2013, at 10:24 PM, Arya Goudarzi <goudarzi@gmail.com> wrote:

> OK. I upgraded one node from 1.1.6 to 1.2.2 today. Despite some new problems that I had
and I posted them in a separate email, this issue still exists but now it is only on 1.2.2
node. This means that the nodes running 1.1.6 see all other nodes including 1.2.2 as Up. Here
is the ring and gossip from nodes with 1.1.6 for example. Bold denotes upgraded node:
> 
> Address         DC          Rack        Status State   Load            Effective-Ownership
Token
>                                                                                     
      141784319550391026443072753098378663700
> XX.180.36    us-east     1b          Up     Normal  49.47 GB        25.00%          
   1808575600
> XX.231.121  us-east     1c          Up     Normal  47.08 GB        25.00%           
  7089215977519551322153637656637080005
> XX.177.177  us-east     1d          Up     Normal  33.64 GB        25.00%           
  14178431955039102644307275311465584410
> XX.7.148    us-east     1b          Up     Normal  41.27 GB        25.00%           
  42535295865117307932921825930779602030
> XX.20.9     us-east     1c          Up     Normal  38.51 GB        25.00%           
  49624511842636859255075463585608106435
> XX.86.255    us-east     1d          Up     Normal  34.78 GB        25.00%          
   56713727820156410577229101240436610840
> XX.63.230    us-east     1b          Up     Normal  38.11 GB        25.00%          
   85070591730234615865843651859750628460
> XX.163.36   us-east     1c          Up     Normal  44.25 GB        25.00%           
  92159807707754167187997289514579132865
> XX.31.234    us-east     1d          Up     Normal  44.66 GB        25.00%          
   99249023685273718510150927169407637270
> XX.132.169   us-east     1b          Up     Normal  44.2 GB         25.00%          
   127605887595351923798765477788721654890
> XX.71.63     us-east     1c          Up     Normal  38.74 GB        25.00%          
   134695103572871475120919115443550159295
> XX.197.209  us-east     1d          Up     Normal  41.5 GB         25.00%           
  141784319550391026443072753098378663700
> 
> /XX.71.63
>   RACK:1c
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.1598705272E10
>   DC:us-east
>   INTERNAL_IP:XX.194.92
>   STATUS:NORMAL,134695103572871475120919115443550159295
>   RPC_ADDRESS:XX.194.92
>   RELEASE_VERSION:1.1.6
> /XX.86.255
>   RACK:1d
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:3.734334162E10
>   DC:us-east
>   INTERNAL_IP:XX.6.195
>   STATUS:NORMAL,56713727820156410577229101240436610840
>   RPC_ADDRESS:XX.6.195
>   RELEASE_VERSION:1.1.6
> /XX.7.148
>   RACK:1b
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.4316975808E10
>   DC:us-east
>   INTERNAL_IP:XX.47.250
>   STATUS:NORMAL,42535295865117307932921825930779602030
>   RPC_ADDRESS:XX.47.250
>   RELEASE_VERSION:1.1.6
> /XX.63.230
>   RACK:1b
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.0918593305E10
>   DC:us-east
>   INTERNAL_IP:XX.89.127
>   STATUS:NORMAL,85070591730234615865843651859750628460
>   RPC_ADDRESS:XX.89.127
>   RELEASE_VERSION:1.1.6
> /XX.132.169
>   RACK:1b
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.745883458E10
>   DC:us-east
>   INTERNAL_IP:XX.94.161
>   STATUS:NORMAL,127605887595351923798765477788721654890
>   RPC_ADDRESS:XX.94.161
>   RELEASE_VERSION:1.1.6
> /XX.180.36
>   RACK:1b
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:5.311963027E10
>   DC:us-east
>   INTERNAL_IP:XX.123.112
>   STATUS:NORMAL,1808575600
>   RPC_ADDRESS:XX.123.112
>   RELEASE_VERSION:1.1.6
> /XX.163.36
>   RACK:1c
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.7516755022E10
>   DC:us-east
>   INTERNAL_IP:XX.163.180
>   STATUS:NORMAL,92159807707754167187997289514579132865
>   RPC_ADDRESS:XX.163.180
>   RELEASE_VERSION:1.1.6
> /XX.31.234
>   RACK:1d
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.7954372912E10
>   DC:us-east
>   INTERNAL_IP:XX.192.159
>   STATUS:NORMAL,99249023685273718510150927169407637270
>   RPC_ADDRESS:XX.192.159
>   RELEASE_VERSION:1.1.6
> /XX.197.209
>   RACK:1d
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.4558968005E10
>   DC:us-east
>   INTERNAL_IP:XX.66.205
>   STATUS:NORMAL,141784319550391026443072753098378663700
>   RPC_ADDRESS:XX.66.205
>   RELEASE_VERSION:1.1.6
> /XX.177.177
>   RACK:1d
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:3.6115572697E10
>   DC:us-east
>   INTERNAL_IP:XX.65.57
>   STATUS:NORMAL,14178431955039102644307275311465584410
>   RPC_ADDRESS:XX.65.57
>   RELEASE_VERSION:1.1.6
> /XX.20.9
>   RACK:1c
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   LOAD:4.1352503882E10
>   DC:us-east
>   INTERNAL_IP:XX.33.229
>   STATUS:NORMAL,49624511842636859255075463585608106435
>   RPC_ADDRESS:XX.33.229
>   RELEASE_VERSION:1.1.6
> /XX.231.121
>   RACK:1c
>   SCHEMA:09487aa5-3380-33ab-b9a5-bcc8476066b0
>   X4:9c765678-d058-4d85-a588-638ce10ff984
>   X3:7
>   DC:us-east
>   INTERNAL_IP:XX.223.241
>   RPC_ADDRESS:XX.223.241
>   RELEASE_VERSION:1.2.2
> 
> Now the nodetool on the 1.2.2 node shows all nodes as Down but itself. Gossipinfo looks
gook though:
> 
> Datacenter: us-east
> ==========
> Replicas: 3
> 
> Address         Rack        Status State   Load            Owns                Token
>                                                                                56713727820156410577229101240436610840
> XX.132.169   1b          Down   Normal  44.2 GB         25.00%              127605887595351923798765477788721654890
> XX.7.148    1b          Down   Normal  41.27 GB        25.00%              42535295865117307932921825930779602030
> XX.180.36    1b          Down   Normal  49.47 GB        25.00%              1808575600
> XX.63.230    1b          Down   Normal  38.11 GB        25.00%              85070591730234615865843651859750628460
> XX.231.121  1c          Up     Normal  47.25 GB        25.00%              7089215977519551322153637656637080005
> XX.71.63     1c          Down   Normal  38.74 GB        25.00%              134695103572871475120919115443550159295
> XX.177.177  1d          Down   Normal  33.64 GB        25.00%              14178431955039102644307275311465584410
> XX.31.234    1d          Down   Normal  44.66 GB        25.00%              99249023685273718510150927169407637270
> XX.20.9     1c          Down   Normal  38.51 GB        25.00%              49624511842636859255075463585608106435
> XX.163.36   1c          Down   Normal  44.25 GB        25.00%              92159807707754167187997289514579132865
> XX.197.209  1d          Down   Normal  41.5 GB         25.00%              141784319550391026443072753098378663700
> XX.86.255    1d          Down   Normal  34.78 GB        25.00%              56713727820156410577229101240436610840
> 
> /XX.71.63
>   RACK:1c
>   RPC_ADDRESS:XX.194.92
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.194.92
>   STATUS:NORMAL,134695103572871475120919115443550159295
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.1598705272E10
> /XX.86.255
>   RACK:1d
>   RPC_ADDRESS:XX.6.195
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.6.195
>   STATUS:NORMAL,56713727820156410577229101240436610840
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:3.7343205002E10
> /XX.7.148
>   RACK:1b
>   RPC_ADDRESS:XX.47.250
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.47.250
>   STATUS:NORMAL,42535295865117307932921825930779602030
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.4316975808E10
> /XX.63.230
>   RACK:1b
>   RPC_ADDRESS:XX.89.127
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.89.127
>   STATUS:NORMAL,85070591730234615865843651859750628460
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.0918456687E10
> /XX.132.169
>   RACK:1b
>   RPC_ADDRESS:XX.94.161
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.94.161
>   STATUS:NORMAL,127605887595351923798765477788721654890
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.745883458E10
> /XX.180.36
>   RACK:1b
>   RPC_ADDRESS:XX.123.112
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.123.112
>   STATUS:NORMAL,1808575600
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:5.311963027E10
> /XX.163.36
>   RACK:1c
>   RPC_ADDRESS:XX.163.180
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.163.180
>   STATUS:NORMAL,92159807707754167187997289514579132865
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.7516755022E10
> /XX.31.234
>   RACK:1d
>   RPC_ADDRESS:XX.192.159
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.192.159
>   STATUS:NORMAL,99249023685273718510150927169407637270
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.7954372912E10
> /XX.197.209
>   RACK:1d
>   RPC_ADDRESS:XX.66.205
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.66.205
>   STATUS:NORMAL,141784319550391026443072753098378663700
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.4559013211E10
> /XX.177.177
>   RACK:1d
>   RPC_ADDRESS:XX.65.57
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.65.57
>   STATUS:NORMAL,14178431955039102644307275311465584410
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:3.6115572697E10
> /XX.20.9
>   RACK:1c
>   RPC_ADDRESS:XX.33.229
>   RELEASE_VERSION:1.1.6
>   INTERNAL_IP:XX.33.229
>   STATUS:NORMAL,49624511842636859255075463585608106435
>   SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
>   DC:us-east
>   LOAD:4.1352367264E10
> /XX.231.121
>   HOST_ID:9c765678-d058-4d85-a588-638ce10ff984
>   RACK:1c
>   RPC_ADDRESS:XX.223.241
>   RELEASE_VERSION:1.2.2
>   INTERNAL_IP:XX.223.241
>   STATUS:NORMAL,7089215977519551322153637656637080005
>   NET_VERSION:7
>   SCHEMA:8b8948f5-d56f-3a96-8005-b9452e42cd67
>   SEVERITY:0.0
>   DC:us-east
>   LOAD:5.0710624207E10
> 
> Is this just a display bug in nodetool or this upgraded node really sees the other ones
as dead?
> 
> -Arya
> 
> 
> On Mon, Feb 25, 2013 at 8:10 PM, Arya Goudarzi <goudarzi@gmail.com> wrote:
> No I did not look at nodetool gossipinfo but from the ring on both pre-upgrade and post
upgrade nodes to 1.2.1, what I observed was the described behavior. 
> 
> 
> On Sat, Feb 23, 2013 at 1:26 AM, Michael Kjellman <mkjellman@barracuda.com> wrote:
> This was a bug with 1.2.0 but resolved in 1.2.1. Did you take a capture of nodetool gossipinfo
and nodetool ring by chance?
> 
> On Feb 23, 2013, at 12:26 AM, "Arya Goudarzi" <goudarzi@gmail.com> wrote:
> 
> > Hi C* users,
> >
> > I just upgrade a 12 node test cluster from 1.1.6 to 1.2.1. What I noticed from nodetool
ring was that the new upgraded nodes only saw each other as Normal and the rest of the cluster
which was on 1.1.6 as Down. Vise versa was true for the nodes running 1.1.6. They saw each
other as Normal but the 1.2.1 nodes as down. I don't see a note in upgrade docs that this
would be an issue. Has anyone else observed this problem?
> >
> > In the debug logs I could see messages saying attempting to connect to node IP and
then saying it is down.
> >
> > Cheers,
> > -Arya
> 
> Copy, by Barracuda, helps you store, protect, and share all your amazing
> 
> things. Start today: www.copy.com.
> 
> 


Mime
View raw message