cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13125) Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9
Date Fri, 20 Jan 2017 14:31:26 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831816#comment-15831816
] 

Sylvain Lebresne commented on CASSANDRA-13125:
----------------------------------------------

Thanks a lot for the very detailed reproduction case and analysis. You are absolutely right
that it's abnormal for 3.0 to transform a range like {{\[c-c!\]\[d-d!\]}} into {{\[c-c!\]\[c-d!\]}}.
And having duplicate row entries is only one of the problem this triggers, as with more collections
you could this ending up to delete data that it shouldn't. In any case, the problem is definitively
in 3.0 generating this in the first place, so that's what we should fix, so fixing this in
LegacyRangeTombstone constructor (Plan-A above) is too late.

So the problem is when building the {{LegacyRangeTombstonList}} on 3.0, though it's not in
the {{insertFrom}} method and the Plan-B patch would likely create other problems. The problem
is in the {{LegacyBoundComparator}}: it delegates to {{clusteringComparator.compare()}} before
it even checks if the bounds it compares have a {{collectionName}}, but that's incorrect.

Anyway, attaching below a fix for this as well as a simple unit test showing
where the problem lies:
| [13125-3.0|https://github.com/pcmanus/cassandra/commits/13125-3.0] | [utests|http://cassci.datastax.com/job/pcmanus-13125-3.0-testall]
| [dtests|http://cassci.datastax.com/job/pcmanus-13125-3.0-dtest] |
| [13125-3.11|https://github.com/pcmanus/cassandra/commits/13125-3.11] | [utests|http://cassci.datastax.com/job/pcmanus-13125-3.11-testall]
| [dtests|http://cassci.datastax.com/job/pcmanus-13125-3.11-dtest] |

[~Yasuharu], if you could check if this patch does properly work in your case, that would
be much appreciated. [~thobbs], since you're familiar with 3.0 compatibility code, would you
also mind having a quick look at the patch?

> Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9
> --------------------------------------------------------
>
>                 Key: CASSANDRA-13125
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13125
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Zhongxiang Zheng
>            Assignee: Yasuharu Goto
>            Priority: Critical
>         Attachments: diff-a.patch, diff-b.patch
>
>
> I found that rows are splitting and duplicated after upgrading the cluster from 2.1.x
to 3.0.x.
> I found the way to reproduce the problem as below.
> {code}
> $ ccm create test -v 2.1.16 -n 3 -s                                                 
                             
> Current cluster is now: test
> $ ccm node1 cqlsh  -e "CREATE KEYSPACE test WITH replication = {'class':'SimpleStrategy',
'replication_factor':3}"
> $ ccm node1 cqlsh -e "CREATE TABLE test.test (id text PRIMARY KEY, value1 set<text>,
value2 set<text>);"
> # Upgrade node1
> $ for i in 1; do ccm node${i} stop; ccm node${i} setdir -v3.0.10; ccm node${i} start;ccm
node${i} nodetool upgradesstables; done
> # Insert a row through node1(3.0.10)
> $ ccm node1 cqlsh -e "INSERT INTO test.test (id, value1, value2) values ('aaa', {'aaa',
'bbb'}, {'ccc', 'ddd'});"                       
> # Insert a row through node2(2.1.16)
> $ ccm node2 cqlsh -e "INSERT INTO test.test (id, value1, value2) values ('bbb', {'aaa',
'bbb'}, {'ccc', 'ddd'});" 
> # The row inserted from node1 is splitting
> $ ccm node1 cqlsh -e "SELECT * FROM test.test ;"
>  id  | value1         | value2
> -----+----------------+----------------
>  aaa |           null |           null
>  aaa | {'aaa', 'bbb'} | {'ccc', 'ddd'}
>  bbb | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> $ for i in 1 2; do ccm node${i} nodetool flush; done
> # Results of sstable2json of node2. The row inserted from node1(3.0.10) is different
from the row inserted from node2(2.1.16).
> $ ccm node2 json -k test -c test
> running
> ['/home/zzheng/.ccm/test/node2/data0/test/test-5406ee80dbdb11e6a175f57c4c7c85f3/test-test-ka-1-Data.db']
> -- test-test-ka-1-Data.db -----
> [
> {"key": "aaa",
>  "cells": [["","",1484564624769577],
>            ["value1","value2:!",1484564624769576,"t",1484564624],
>            ["value1:616161","",1484564624769577],
>            ["value1:626262","",1484564624769577],
>            ["value2:636363","",1484564624769577],
>            ["value2:646464","",1484564624769577]]},
> {"key": "bbb",
>  "cells": [["","",1484564634508029],
>            ["value1:_","value1:!",1484564634508028,"t",1484564634],
>            ["value1:616161","",1484564634508029],
>            ["value1:626262","",1484564634508029],
>            ["value2:_","value2:!",1484564634508028,"t",1484564634],
>            ["value2:636363","",1484564634508029],
>            ["value2:646464","",1484564634508029]]}
> ]
> # Upgrade node2,3
> $ for i in `seq 2 3`; do ccm node${i} stop; ccm node${i} setdir -v3.0.10; ccm node${i}
start;ccm node${i} nodetool upgradesstables; done
> # After upgrade node2,3, the row inserted from node1 is splitting in node2,3
> $ ccm node2 cqlsh -e "SELECT * FROM test.test ;"                                    
                                                   
>  id  | value1         | value2
> -----+----------------+----------------
>  aaa |           null |           null
>  aaa | {'aaa', 'bbb'} | {'ccc', 'ddd'}
>  bbb | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> (3 rows)
> # Results of sstabledump
> # node1
> [
>   {
>     "partition" : {
>       "key" : [ "aaa" ],
>       "position" : 0
>     },
>     "rows" : [
>       {
>         "type" : "row",
>         "position" : 17,
>         "liveness_info" : { "tstamp" : "2017-01-16T11:03:44.769577Z" },
>         "cells" : [
>           { "name" : "value1", "deletion_info" : { "marked_deleted" : "2017-01-16T11:03:44.769576Z",
"local_delete_time" : "2017-01-16T11:03:44Z" } },
>           { "name" : "value1", "path" : [ "aaa" ], "value" : "" },
>           { "name" : "value1", "path" : [ "bbb" ], "value" : "" },
>           { "name" : "value2", "deletion_info" : { "marked_deleted" : "2017-01-16T11:03:44.769576Z",
"local_delete_time" : "2017-01-16T11:03:44Z" } },
>           { "name" : "value2", "path" : [ "ccc" ], "value" : "" },
>           { "name" : "value2", "path" : [ "ddd" ], "value" : "" }
>         ]
>       }
>     ]
>   },
>   {
>     "partition" : {
>       "key" : [ "bbb" ],
>       "position" : 48
>     },
>     "rows" : [
>       {
>         "type" : "row",
>         "position" : 65,
>         "liveness_info" : { "tstamp" : "2017-01-16T11:03:54.508029Z" },
>         "cells" : [
>           { "name" : "value1", "deletion_info" : { "marked_deleted" : "2017-01-16T11:03:54.508028Z",
"local_delete_time" : "2017-01-16T11:03:54Z" } },
>           { "name" : "value1", "path" : [ "aaa" ], "value" : "" },
>           { "name" : "value1", "path" : [ "bbb" ], "value" : "" },
>           { "name" : "value2", "deletion_info" : { "marked_deleted" : "2017-01-16T11:03:54.508028Z",
"local_delete_time" : "2017-01-16T11:03:54Z" } },
>           { "name" : "value2", "path" : [ "ccc" ], "value" : "" },
>           { "name" : "value2", "path" : [ "ddd" ], "value" : "" }
>         ]
>       }
>     ]
>   }
> ]                                                                                   
                                                                
> # node2
> [
>   {
>     "partition" : {
>       "key" : [ "aaa" ],
>       "position" : 0
>     },
>     "rows" : [
>       {
>         "type" : "row",
>         "position" : 17,
>         "liveness_info" : { "tstamp" : "2017-01-16T11:03:44.769577Z" },
>         "cells" : [ ]
>       },
>       {
>         "type" : "row",
>         "position" : 22,
>         "deletion_info" : { "marked_deleted" : "2017-01-16T11:03:44.769576Z", "local_delete_time"
: "2017-01-16T11:03:44Z" },
>         "cells" : [
>           { "name" : "value1", "path" : [ "aaa" ], "value" : "", "tstamp" : "2017-01-16T11:03:44.769577Z"
},
>           { "name" : "value1", "path" : [ "bbb" ], "value" : "", "tstamp" : "2017-01-16T11:03:44.769577Z"
},
>           { "name" : "value2", "path" : [ "ccc" ], "value" : "", "tstamp" : "2017-01-16T11:03:44.769577Z"
},
>           { "name" : "value2", "path" : [ "ddd" ], "value" : "", "tstamp" : "2017-01-16T11:03:44.769577Z"
}
>         ]
>       }
>     ]
>   },
>   {
>     "partition" : {
>       "key" : [ "bbb" ],
>       "position" : 57
>     },
>     "rows" : [
>       {
>         "type" : "row",
>         "position" : 74,
>         "liveness_info" : { "tstamp" : "2017-01-16T11:03:54.508029Z" },
>         "cells" : [
>           { "name" : "value1", "deletion_info" : { "marked_deleted" : "2017-01-16T11:03:54.508028Z",
"local_delete_time" : "2017-01-16T11:03:54Z" } },
>           { "name" : "value1", "path" : [ "aaa" ], "value" : "" },
>           { "name" : "value1", "path" : [ "bbb" ], "value" : "" },
>           { "name" : "value2", "deletion_info" : { "marked_deleted" : "2017-01-16T11:03:54.508028Z",
"local_delete_time" : "2017-01-16T11:03:54Z" } },
>           { "name" : "value2", "path" : [ "ccc" ], "value" : "" },
>           { "name" : "value2", "path" : [ "ddd" ], "value" : "" }
>         ]
>       }
>     ]
>   }
> ]
> {code}
> Another example of row splitting is as follows.
> {code}
> $ ccm create test2 -v 2.1.16 -n 3 -s                                                
                                                   
> Current cluster is now: test2
> $ ccm node1 cqlsh  -e "CREATE KEYSPACE test WITH replication = {'class':'SimpleStrategy',
'replication_factor':3}"                      
> $ ccm node1 cqlsh -e "CREATE TABLE test.text_set_set (id text PRIMARY KEY, value1 text,
value2 set<text>, value3 set<text>);"           
> $ for i in `seq 1`; do ccm node${i} stop; ccm node${i} setdir -v3.0.10; ccm node${i}
start;ccm node${i} nodetool upgradesstables; done  
> $ ccm node1 cqlsh -e "INSERT INTO test.text_set_set (id, value1, value2, value3) values
('aaa', 'aaa', {'aaa', 'bbb'}, {'ccc', 'ddd'});"
> $ ccm node1 cqlsh -e "SELECT * FROM test.text_set_set;"                             
                                                   
>  id  | value1 | value2         | value3
> -----+--------+----------------+----------------
>  aaa |    aaa |           null |           null
>  aaa |   null | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> (2 rows)
> {code}
> As far as I investigated, the occurrence conditions are as follows.
> * Table schema contains multiple collections.
> * Insert a row, which values of the collection column are not null through 3.x node while
both 2.1 and 3.x nodes exist in a cluster.
> * Rows in sstables of node which version was 2.1 at the time the row was inserted is
splitting after upgrading to 3.x.
> Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message