cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-6554) During upgrade from 1.2 -> 2.0, upgraded node sees other nodes as Down
Date Thu, 09 Jan 2014 18:57:52 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13866905#comment-13866905
] 

Brandon Williams edited comment on CASSANDRA-6554 at 1/9/14 6:56 PM:
---------------------------------------------------------------------

What I'm seeing is that on the first boot of 2.0, ring/status show the other nodes as down,
even though the logs indicate everything is fine.  And indeed, it is, reads and writes work
normally.  The other nodes on 1.2 see the 2.0 node as up.  If you restart the 2.0 node at
this point, the cosmetic problem goes away and everything appears normal.  This appears to
stem from CASSANDRA-3533 unconditionally sending echo commands and only adding the node to
liveMembers if it responds, which cannot happen with nodes < 2.0.  Patch to conditionally
mark anything < 2.0 up without the echo.


was (Author: brandon.williams):
What I'm seeing is that on the first boot of 2.0, ring/status show the other nodes as down,
even though the logs indicate everything is fine.  And indeed, it is, reads and writes work
normally.  The other nodes on 1.2 see the 2.0 node as up.  If you restart the 2.0 node at
this point, the cosmetic problem goes away and everything appears normal.  This appears to
stem from CASSANDRA-3554 unconditionally sending echo commands and only adding the node to
liveMembers if it responds, which cannot happen with nodes < 2.0.  Patch to conditionally
mark anything < 2.0 up without the echo.

> During upgrade from 1.2 -> 2.0, upgraded node sees other nodes as Down
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-6554
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6554
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: EC2 Ubuntu Precise 12.04
> Oracle JRE 1.7_25
> C* 1.2.13 upgrade to 2.0.4
>            Reporter: Michael Shuler
>            Assignee: Brandon Williams
>             Fix For: 2.0.5
>
>         Attachments: 6554.txt, 6554_trace_system.log, 6554_trace_system.node1.log, 6554_trace_system.node2.log
>
>
> During an upgrade from 1.2.13 to 2.0.3/2.0.4, the upgraded node sees the remaining nodes
of the cluster as Down.
> {code}
> automaton@ip-10-139-1-113:~$ nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address         Load       Owns   Host ID                               Token   
                                Rack
> UN  10.139.1.113    98.94 MB   33.3%  33b1cd06-e17b-4332-8066-0c6c401e0cf3  -9223372036854775808
                    rack1
> DN  10.139.11.168   97.51 MB   33.3%  ec97c163-8f2d-4019-a3d1-55df5e4037d4  -3074457345618258603
                    rack1
> DN  10.238.221.115  97.34 MB   33.3%  73a76d3f-73ef-481d-b603-0833c0ff80c2  3074457345618258602
                     rack1
> automaton@ip-10-139-1-113:~$ nodetool gossipinfo
> /10.238.221.115
>   SEVERITY:0.0
>   RPC_ADDRESS:0.0.0.0
>   DC:datacenter1
>   RELEASE_VERSION:1.2.13
>   LOAD:1.02066255E8
>   STATUS:NORMAL,3074457345618258602
>   SCHEMA:8b351435-81ef-3a14-adf7-8555e2f19ecd
>   NET_VERSION:6
>   RACK:rack1
>   HOST_ID:73a76d3f-73ef-481d-b603-0833c0ff80c2
> /10.139.1.113
>   RPC_ADDRESS:0.0.0.0
>   SEVERITY:0.0
>   DC:datacenter1
>   RELEASE_VERSION:2.0.4
>   LOAD:1.03750451E8
>   STATUS:NORMAL,-9223372036854775808
>   SCHEMA:dfafb212-5b8f-31cb-a80b-2ba58fcef73d
>   NET_VERSION:7
>   RACK:rack1
>   HOST_ID:33b1cd06-e17b-4332-8066-0c6c401e0cf3
> /10.139.11.168
>   SEVERITY:0.0
>   RPC_ADDRESS:0.0.0.0
>   DC:datacenter1
>   RELEASE_VERSION:1.2.13
>   LOAD:1.02245066E8
>   STATUS:NORMAL,-3074457345618258603
>   SCHEMA:8b351435-81ef-3a14-adf7-8555e2f19ecd
>   NET_VERSION:6
>   RACK:rack1
>   HOST_ID:ec97c163-8f2d-4019-a3d1-55df5e4037d4
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message