incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arya Goudarzi <gouda...@gmail.com>
Subject Re: Incompatible Gossip 1.1.6 to 1.2.1 Upgrade?
Date Sat, 09 Mar 2013 06:24:54 GMT
OK. I upgraded one node from 1.1.6 to 1.2.2 today. Despite some new
problems that I had and I posted them in a separate email, this issue still
exists but now it is only on 1.2.2 node. This means that the nodes running
1.1.6 see all other nodes including 1.2.2 as Up. Here is the ring and
gossip from nodes with 1.1.6 for example. Bold denotes upgraded node:

Address         DC          Rack        Status State   Load
 Effective-Ownership Token

               141784319550391026443072753098378663700
XX.180.36    us-east     1b          Up     Normal  49.47 GB        25.00%
             1808575600
*XX.231.121  us-east     1c          Up     Normal  47.08 GB        25.00%
             7089215977519551322153637656637080005*
XX.177.177  us-east     1d          Up     Normal  33.64 GB        25.00%
           14178431955039102644307275311465584410
XX.7.148    us-east     1b          Up     Normal  41.27 GB        25.00%
           42535295865117307932921825930779602030
XX.20.9     us-east     1c          Up     Normal  38.51 GB        25.00%
           49624511842636859255075463585608106435
XX.86.255    us-east     1d          Up     Normal  34.78 GB        25.00%
             56713727820156410577229101240436610840
XX.63.230    us-east     1b          Up     Normal  38.11 GB        25.00%
             85070591730234615865843651859750628460
XX.163.36   us-east     1c          Up     Normal  44.25 GB        25.00%
           92159807707754167187997289514579132865
XX.31.234    us-east     1d          Up     Normal  44.66 GB        25.00%
             99249023685273718510150927169407637270
XX.132.169   us-east     1b          Up     Normal  44.2 GB         25.00%
             127605887595351923798765477788721654890
XX.71.63     us-east     1c          Up     Normal  38.74 GB        25.00%
             134695103572871475120919115443550159295
XX.197.209  us-east     1d          Up     Normal  41.5 GB         25.00%
           141784319550391026443072753098378663700

/XX.71.63
  RACK:1c
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:4.1598705272E10
  DC:us-east
  INTERNAL_IP:XX.194.92
  STATUS:NORMAL,134695103572871475120919115443550159295
  RPC_ADDRESS:XX.194.92
  RELEASE_VERSION:1.1.6
/XX.86.255
  RACK:1d
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:3.734334162E10
  DC:us-east
  INTERNAL_IP:XX.6.195
  STATUS:NORMAL,56713727820156410577229101240436610840
  RPC_ADDRESS:XX.6.195
  RELEASE_VERSION:1.1.6
/XX.7.148
  RACK:1b
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:4.4316975808E10
  DC:us-east
  INTERNAL_IP:XX.47.250
  STATUS:NORMAL,42535295865117307932921825930779602030
  RPC_ADDRESS:XX.47.250
  RELEASE_VERSION:1.1.6
/XX.63.230
  RACK:1b
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:4.0918593305E10
  DC:us-east
  INTERNAL_IP:XX.89.127
  STATUS:NORMAL,85070591730234615865843651859750628460
  RPC_ADDRESS:XX.89.127
  RELEASE_VERSION:1.1.6
/XX.132.169
  RACK:1b
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:4.745883458E10
  DC:us-east
  INTERNAL_IP:XX.94.161
  STATUS:NORMAL,127605887595351923798765477788721654890
  RPC_ADDRESS:XX.94.161
  RELEASE_VERSION:1.1.6
/XX.180.36
  RACK:1b
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:5.311963027E10
  DC:us-east
  INTERNAL_IP:XX.123.112
  STATUS:NORMAL,1808575600
  RPC_ADDRESS:XX.123.112
  RELEASE_VERSION:1.1.6
/XX.163.36
  RACK:1c
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:4.7516755022E10
  DC:us-east
  INTERNAL_IP:XX.163.180
  STATUS:NORMAL,92159807707754167187997289514579132865
  RPC_ADDRESS:XX.163.180
  RELEASE_VERSION:1.1.6
/XX.31.234
  RACK:1d
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:4.7954372912E10
  DC:us-east
  INTERNAL_IP:XX.192.159
  STATUS:NORMAL,99249023685273718510150927169407637270
  RPC_ADDRESS:XX.192.159
  RELEASE_VERSION:1.1.6
/XX.197.209
  RACK:1d
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:4.4558968005E10
  DC:us-east
  INTERNAL_IP:XX.66.205
  STATUS:NORMAL,141784319550391026443072753098378663700
  RPC_ADDRESS:XX.66.205
  RELEASE_VERSION:1.1.6
/XX.177.177
  RACK:1d
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:3.6115572697E10
  DC:us-east
  INTERNAL_IP:XX.65.57
  STATUS:NORMAL,14178431955039102644307275311465584410
  RPC_ADDRESS:XX.65.57
  RELEASE_VERSION:1.1.6
/XX.20.9
  RACK:1c
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  LOAD:4.1352503882E10
  DC:us-east
  INTERNAL_IP:XX.33.229
  STATUS:NORMAL,49624511842636859255075463585608106435
  RPC_ADDRESS:XX.33.229
  RELEASE_VERSION:1.1.6
*/XX.231.121*
*  RACK:1c*
*  SCHEMA:09487aa5-3380-33ab-b9a5-bcc8476066b0*
*  X4:9c765678-d058-4d85-a588-638ce10ff984*
*  X3:7*
*  DC:us-east*
*  INTERNAL_IP:XX.223.241*
*  RPC_ADDRESS:XX.223.241*
*  RELEASE_VERSION:1.2.2*

Now the nodetool on the 1.2.2 node shows all nodes as Down but itself.
Gossipinfo looks gook though:

Datacenter: us-east
==========
Replicas: 3

Address         Rack        Status State   Load            Owns
   Token

   56713727820156410577229101240436610840
XX.132.169   1b          Down   Normal  44.2 GB         25.00%
 127605887595351923798765477788721654890
XX.7.148    1b          Down   Normal  41.27 GB        25.00%
 42535295865117307932921825930779602030
XX.180.36    1b          Down   Normal  49.47 GB        25.00%
 1808575600
XX.63.230    1b          Down   Normal  38.11 GB        25.00%
 85070591730234615865843651859750628460
*XX.231.121  1c          Up     Normal  47.25 GB        25.00%
 7089215977519551322153637656637080005*
XX.71.63     1c          Down   Normal  38.74 GB        25.00%
 134695103572871475120919115443550159295
XX.177.177  1d          Down   Normal  33.64 GB        25.00%
 14178431955039102644307275311465584410
XX.31.234    1d          Down   Normal  44.66 GB        25.00%
 99249023685273718510150927169407637270
XX.20.9     1c          Down   Normal  38.51 GB        25.00%
 49624511842636859255075463585608106435
XX.163.36   1c          Down   Normal  44.25 GB        25.00%
 92159807707754167187997289514579132865
XX.197.209  1d          Down   Normal  41.5 GB         25.00%
 141784319550391026443072753098378663700
XX.86.255    1d          Down   Normal  34.78 GB        25.00%
 56713727820156410577229101240436610840

/XX.71.63
  RACK:1c
  RPC_ADDRESS:XX.194.92
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.194.92
  STATUS:NORMAL,134695103572871475120919115443550159295
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:4.1598705272E10
/XX.86.255
  RACK:1d
  RPC_ADDRESS:XX.6.195
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.6.195
  STATUS:NORMAL,56713727820156410577229101240436610840
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:3.7343205002E10
/XX.7.148
  RACK:1b
  RPC_ADDRESS:XX.47.250
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.47.250
  STATUS:NORMAL,42535295865117307932921825930779602030
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:4.4316975808E10
/XX.63.230
  RACK:1b
  RPC_ADDRESS:XX.89.127
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.89.127
  STATUS:NORMAL,85070591730234615865843651859750628460
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:4.0918456687E10
/XX.132.169
  RACK:1b
  RPC_ADDRESS:XX.94.161
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.94.161
  STATUS:NORMAL,127605887595351923798765477788721654890
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:4.745883458E10
/XX.180.36
  RACK:1b
  RPC_ADDRESS:XX.123.112
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.123.112
  STATUS:NORMAL,1808575600
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:5.311963027E10
/XX.163.36
  RACK:1c
  RPC_ADDRESS:XX.163.180
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.163.180
  STATUS:NORMAL,92159807707754167187997289514579132865
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:4.7516755022E10
/XX.31.234
  RACK:1d
  RPC_ADDRESS:XX.192.159
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.192.159
  STATUS:NORMAL,99249023685273718510150927169407637270
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:4.7954372912E10
/XX.197.209
  RACK:1d
  RPC_ADDRESS:XX.66.205
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.66.205
  STATUS:NORMAL,141784319550391026443072753098378663700
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:4.4559013211E10
/XX.177.177
  RACK:1d
  RPC_ADDRESS:XX.65.57
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.65.57
  STATUS:NORMAL,14178431955039102644307275311465584410
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:3.6115572697E10
/XX.20.9
  RACK:1c
  RPC_ADDRESS:XX.33.229
  RELEASE_VERSION:1.1.6
  INTERNAL_IP:XX.33.229
  STATUS:NORMAL,49624511842636859255075463585608106435
  SCHEMA:99dce53b-487e-3e7b-a958-a1cc48d9f575
  DC:us-east
  LOAD:4.1352367264E10
*/XX.231.121*
*  HOST_ID:9c765678-d058-4d85-a588-638ce10ff984*
*  RACK:1c*
*  RPC_ADDRESS:XX.223.241*
*  RELEASE_VERSION:1.2.2*
*  INTERNAL_IP:XX.223.241*
*  STATUS:NORMAL,7089215977519551322153637656637080005*
*  NET_VERSION:7*
*  SCHEMA:8b8948f5-d56f-3a96-8005-b9452e42cd67*
*  SEVERITY:0.0*
*  DC:us-east*
*  LOAD:5.0710624207E10*

Is this just a display bug in nodetool or this upgraded node really sees
the other ones as dead?

-Arya


On Mon, Feb 25, 2013 at 8:10 PM, Arya Goudarzi <goudarzi@gmail.com> wrote:

> No I did not look at nodetool gossipinfo but from the ring on both
> pre-upgrade and post upgrade nodes to 1.2.1, what I observed was the
> described behavior.
>
>
> On Sat, Feb 23, 2013 at 1:26 AM, Michael Kjellman <mkjellman@barracuda.com
> > wrote:
>
>> This was a bug with 1.2.0 but resolved in 1.2.1. Did you take a capture
>> of nodetool gossipinfo and nodetool ring by chance?
>>
>> On Feb 23, 2013, at 12:26 AM, "Arya Goudarzi" <goudarzi@gmail.com> wrote:
>>
>> > Hi C* users,
>> >
>> > I just upgrade a 12 node test cluster from 1.1.6 to 1.2.1. What I
>> noticed from nodetool ring was that the new upgraded nodes only saw each
>> other as Normal and the rest of the cluster which was on 1.1.6 as Down.
>> Vise versa was true for the nodes running 1.1.6. They saw each other as
>> Normal but the 1.2.1 nodes as down. I don't see a note in upgrade docs that
>> this would be an issue. Has anyone else observed this problem?
>> >
>> > In the debug logs I could see messages saying attempting to connect to
>> node IP and then saying it is down.
>> >
>> > Cheers,
>> > -Arya
>>
>> Copy, by Barracuda, helps you store, protect, and share all your amazing
>>
>> things. Start today: www.copy.com.
>>
>
>

Mime
View raw message