cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Shuler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6554) Cluster is read-only during upgrade of nodes from 1.2 -> 2.0
Date Tue, 07 Jan 2014 18:16:04 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13864489#comment-13864489
] 

Michael Shuler commented on CASSANDRA-6554:
-------------------------------------------

Aha!  Running stress with the whole cluster online, prior to upgrading node0 (above), meant
that all nodes had Keyspace1.

The write failure that I believe both of us were seeing last night was running stress on the
upgraded node and other nodes without ever having run it prior, so the schemas don't agree
and cannot propogate the keyspace:

node0 (upgraded)
{code}
automaton@ip-10-180-239-135:~$ ./dsc-cassandra/tools/bin/cassandra-stress
Created keyspaces. Sleeping 1s for propagation.
total,interval_op_rate,interval_key_rate,latency,95th,99.9th,elapsed_time
25,2,2,238.9,263.2,263.3,10
25,0,0,238.9,263.2,263.3,20
Operation [26] retried 10 times - error inserting key 0000026 ((TimedOutException))

Operation [19] retried 10 times - error inserting key 0000019 ((TimedOutException))

Operation [18] retried 10 times - error inserting key 0000018 ((TimedOutException))

Operation [3] retried 10 times - error inserting key 0000003 ((TimedOutException))

Operation [46] retried 10 times - error inserting key 0000046 ((TimedOutException))

Operation [47] retried 10 times - error inserting key 0000047 ((TimedOutException))

Operation [15] retried 10 times - error inserting key 0000015 ((TimedOutException))

Operation [6] retried 10 times - error inserting key 0000006 ((TimedOutException))

Operation [2] retried 10 times - error inserting key 0000002 ((TimedOutException))

Operation [37] retried 10 times - error inserting key 0000037 ((TimedOutException))

Operation [14] retried 10 times - error inserting key 0000014 ((TimedOutException))

Operation [41] retried 10 times - error inserting key 0000041 ((TimedOutException))

Operation [39] retried 10 times - error inserting key 0000039 ((TimedOutException))

Operation [22] retried 10 times - error inserting key 0000022 ((TimedOutException))

Operation [0] retried 10 times - error inserting key 0000000 ((TimedOutException))

Operation [16] retried 10 times - error inserting key 0000016 ((TimedOutException))

Operation [45] retried 10 times - error inserting key 0000045 ((TimedOutException))

Operation [12] retried 10 times - error inserting key 0000012 ((TimedOutException))

Operation [32] retried 10 times - error inserting key 0000032 ((TimedOutException))

Operation [9] retried 10 times - error inserting key 0000009 ((TimedOutException))

Operation [10] retried 10 times - error inserting key 0000010 ((TimedOutException))

Operation [44] retried 10 times - error inserting key 0000044 ((TimedOutException))

Operation [34] retried 10 times - error inserting key 0000034 ((TimedOutException))

Operation [5] retried 10 times - error inserting key 0000005 ((TimedOutException))

Operation [49] retried 10 times - error inserting key 0000049 ((TimedOutException))

Operation [11] retried 10 times - error inserting key 0000011 ((TimedOutException))

Operation [54] retried 10 times - error inserting key 0000054 ((TimedOutException))

Operation [33] retried 10 times - error inserting key 0000033 ((TimedOutException))

Operation [51] retried 10 times - error inserting key 0000051 ((TimedOutException))

Operation [57] retried 10 times - error inserting key 0000057 ((TimedOutException))

Operation [58] retried 10 times - error inserting key 0000058 ((TimedOutException))

Operation [53] retried 10 times - error inserting key 0000053 ((TimedOutException))

Operation [55] retried 10 times - error inserting key 0000055 ((TimedOutException))

Operation [8] retried 10 times - error inserting key 0000008 ((TimedOutException))

Operation [42] retried 10 times - error inserting key 0000042 ((TimedOutException))

Operation [52] retried 10 times - error inserting key 0000052 ((TimedOutException))

Operation [20] retried 10 times - error inserting key 0000020 ((TimedOutException))

Operation [31] retried 10 times - error inserting key 0000031 ((TimedOutException))

Operation [63] retried 10 times - error inserting key 0000063 ((TimedOutException))

Operation [59] retried 10 times - error inserting key 0000059 ((TimedOutException))

Operation [64] retried 10 times - error inserting key 0000064 ((TimedOutException))

Operation [66] retried 10 times - error inserting key 0000066 ((TimedOutException))

Operation [61] retried 10 times - error inserting key 0000061 ((TimedOutException))

Operation [67] retried 10 times - error inserting key 0000067 ((TimedOutException))

Operation [60] retried 10 times - error inserting key 0000060 ((TimedOutException))

Operation [70] retried 10 times - error inserting key 0000070 ((TimedOutException))

Operation [71] retried 10 times - error inserting key 0000071 ((TimedOutException))

Operation [73] retried 10 times - error inserting key 0000073 ((TimedOutException))

Operation [72] retried 10 times - error inserting key 0000072 ((TimedOutException))

Operation [74] retried 10 times - error inserting key 0000074 ((TimedOutException))

25,0,0,238.9,263.2,263.3,21
FAILURE
automaton@ip-10-180-239-135:~$
{code}

node1 and node2 error the same way:
{code}
automaton@ip-10-180-230-58:~$ ./dsc-cassandra/tools/bin/cassandra-stress
Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("Keyspace1"
conflicts with "Keyspace1")
total,interval_op_rate,interval_key_rate,latency/95th/99.9th,elapsed_time
Operation [9] retried 10 times - error inserting key 0000009 ((InvalidRequestException): unconfigured
columnfamily Standard1)

Operation [31] retried 10 times - error inserting key 0000031 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [27] retried 10 times - error inserting key 0000027 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [46] retried 10 times - error inserting key 0000046 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [11] retried 10 times - error inserting key 0000011 ((InvalidRequestException):
unconfigured columnfamily Standard1)                                                     
                                        
                                                                                         
                    
Operation [39] retried 10 times - error inserting key 0000039 ((InvalidRequestException):
unconfigured columnfamily Standard1)                                                     
                                        
                                                                                         
                    
Operation [43] retried 10 times - error inserting key 0000043 ((InvalidRequestException):
unconfigured columnfamily Standard1)                                                     
                                        
                                                                                         
                    
Operation [2] retried 10 times - error inserting key 0000002 ((InvalidRequestException): unconfigured
columnfamily Standard1)                                                                  
                            

Operation [13] retried 10 times - error inserting key 0000013 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [37] retried 10 times - error inserting key 0000037 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [49] retried 10 times - error inserting key 0000049 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [15] retried 10 times - error inserting key 0000015 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [19] retried 10 times - error inserting key 0000019 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [28] retried 10 times - error inserting key 0000028 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [21] retried 10 times - error inserting key 0000021 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [10] retried 10 times - error inserting key 0000010 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [17] retried 10 times - error inserting key 0000017 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [18] retried 10 times - error inserting key 0000018 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [30] retried 10 times - error inserting key 0000030 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [7] retried 10 times - error inserting key 0000007 ((InvalidRequestException): unconfigured
columnfamily Standard1)

Operation [42] retried 10 times - error inserting key 0000042 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [6] retried 10 times - error inserting key 0000006 ((InvalidRequestException): unconfigured
columnfamily Standard1)

Operation [25] retried 10 times - error inserting key 0000025 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [5] retried 10 times - error inserting key 0000005 ((InvalidRequestException): unconfigured
columnfamily Standard1)

Operation [44] retried 10 times - error inserting key 0000044 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [41] retried 10 times - error inserting key 0000041 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [16] retried 10 times - error inserting key 0000016 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [36] retried 10 times - error inserting key 0000036 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [14] retried 10 times - error inserting key 0000014 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [23] retried 10 times - error inserting key 0000023 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [29] retried 10 times - error inserting key 0000029 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [40] retried 10 times - error inserting key 0000040 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [1] retried 10 times - error inserting key 0000001 ((InvalidRequestException): unconfigured
columnfamily Standard1)

Operation [24] retried 10 times - error inserting key 0000024 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [8] retried 10 times - error inserting key 0000008 ((InvalidRequestException): unconfigured
columnfamily Standard1)

Operation [32] retried 10 times - error inserting key 0000032 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [4] retried 10 times - error inserting key 0000004 ((InvalidRequestException): unconfigured
columnfamily Standard1)

Operation [33] retried 10 times - error inserting key 0000033 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [48] retried 10 times - error inserting key 0000048 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [35] retried 10 times - error inserting key 0000035 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [45] retried 10 times - error inserting key 0000045 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [22] retried 10 times - error inserting key 0000022 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [34] retried 10 times - error inserting key 0000034 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [26] retried 10 times - error inserting key 0000026 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [47] retried 10 times - error inserting key 0000047 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [3] retried 10 times - error inserting key 0000003 ((InvalidRequestException): unconfigured
columnfamily Standard1)

Operation [20] retried 10 times - error inserting key 0000020 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [12] retried 10 times - error inserting key 0000012 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [38] retried 10 times - error inserting key 0000038 ((InvalidRequestException):
unconfigured columnfamily Standard1)

Operation [0] retried 10 times - error inserting key 0000000 ((InvalidRequestException): unconfigured
columnfamily Standard1)

0,0,0,0.0,0.0,0.0,1
FAILURE
automaton@ip-10-180-230-58:~$
{code}

I believe the answer to this is notabug and "Don't do that."  :)

> Cluster is read-only during upgrade of nodes from 1.2 -> 2.0
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-6554
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6554
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: EC2 Ubuntu Precise 12.04
> Oracle JRE 1.7_25
> C* 1.2.13 upgrade to 2.0.4
>            Reporter: Michael Shuler
>
> During an upgrade from 1.2.13 to 2.0.3/2.0.4, the cluster is read-only and writes fail,
until the entire cluster is fully upgraded.
> (I'm gathering complete repro steps, test results, and logs to try to help and will post
those asap, as well as try other versions to see what happens)



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message