Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 704A7200CED for ; Fri, 18 Aug 2017 14:14:07 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 3F7EC16CAE2; Fri, 18 Aug 2017 12:14:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 357BD16CAE0 for ; Fri, 18 Aug 2017 14:14:06 +0200 (CEST) Received: (qmail 70711 invoked by uid 500); 18 Aug 2017 12:14:04 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 70700 invoked by uid 99); 18 Aug 2017 12:14:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Aug 2017 12:14:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D5124180582 for ; Fri, 18 Aug 2017 12:14:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id hsC0RdhmkyXI for ; Fri, 18 Aug 2017 12:14:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 77F795F659 for ; Fri, 18 Aug 2017 12:14:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 80573E0059 for ; Fri, 18 Aug 2017 12:14:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 25E9021409 for ; Fri, 18 Aug 2017 12:14:00 +0000 (UTC) Date: Fri, 18 Aug 2017 12:14:00 +0000 (UTC) From: "Benjamin Lerer (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-13776) Adding a field to an UDT can corrupte the tables using it MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 18 Aug 2017 12:14:07 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-13776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-13776: --------------------------------------- Description: Adding a field to an UDT which is used as a {{Set}} element or as a {{Map}} element can corrupt the table. The problem can be reproduced using the following test case: {code} @Test public void testReadAfterAlteringUserTypeNestedWithinSet() throws Throwable { String ut1 = createType("CREATE TYPE %s (a int)"); String columnType = KEYSPACE + "." + ut1; try { createTable("CREATE TABLE %s (x int PRIMARY KEY, y set>)"); disableCompaction(); execute("INSERT INTO %s (x, y) VALUES(1, ?)", set(userType(1), userType(2))); assertRows(execute("SELECT * FROM %s"), row(1, set(userType(1), userType(2)))); flush(); assertRows(execute("SELECT * FROM %s WHERE x = 1"), row(1, set(userType(1), userType(2)))); execute("ALTER TYPE " + KEYSPACE + "." + ut1 + " ADD b int"); execute("UPDATE %s SET y = y + ? WHERE x = 1", set(userType(1, 1), userType(1, 2), userType(2, 1))); flush(); assertRows(execute("SELECT * FROM %s WHERE x = 1"), row(1, set(userType(1), userType(1, 1), userType(1, 2), userType(2), userType(2, 1)))); compact(); assertRows(execute("SELECT * FROM %s WHERE x = 1"), row(1, set(userType(1), userType(1, 1), userType(1, 2), userType(2), userType(2, 1)))); } finally { enableCompaction(); } } {code} There are in fact 2 problems: # When the {{sets}} from the 2 versions are merged the {{ColumnDefinition}} being picked up can be the older one. In which case when the tuples are sorted it my lead to an {{IndexOutOfBoundsException}}. # During compaction, the old column definition can be the one being kept for the SSTable metadata. If it is the case the SSTable will not be readable any more and will be marked as {{corrupted}}. If one of the tables using the type has a Materialized View attached to it, the MV updates can also fail with {{IndexOutOfBoundsException}}. This problem can be reproduced using the following test: {code} @Test public void testAlteringUserTypeNestedWithinSetWithView() throws Throwable { String columnType = typeWithKs(createType("CREATE TYPE %s (a int)")); createTable("CREATE TABLE %s (pk int, c int, v int, s set>, PRIMARY KEY (pk, c))"); execute("CREATE MATERIALIZED VIEW " + keyspace() + ".view1 AS SELECT c, pk, v FROM %s WHERE pk IS NOT NULL AND c IS NOT NULL AND v IS NOT NULL PRIMARY KEY (c, pk)"); execute("INSERT INTO %s (pk, c, v, s) VALUES(?, ?, ?, ?)", 1, 1, 1, set(userType(1), userType(2))); flush(); execute("ALTER TYPE " + columnType + " ADD b int"); execute("UPDATE %s SET s = s + ?, v = ? WHERE pk = ? AND c = ?", set(userType(1, 1), userType(1, 2), userType(2, 1)), 2, 1, 1); assertRows(execute("SELECT * FROM %s WHERE pk = ? AND c = ?", 1, 1), row(1, 1, 2, set(userType(1), userType(1, 1), userType(1, 2), userType(2), userType(2, 1)))); } {code} was: Adding a field to an UDT which is used as a {{Set}} element or as a {{Map}} element can corrupt the table. The problem can be reproduced using the following test case: {code} @Test public void testReadAfterAlteringUserTypeNestedWithinSet() throws Throwable { String ut1 = createType("CREATE TYPE %s (a int)"); String columnType = KEYSPACE + "." + ut1; try { createTable("CREATE TABLE %s (x int PRIMARY KEY, y set>)"); disableCompaction(); execute("INSERT INTO %s (x, y) VALUES(1, ?)", set(userType(1), userType(2))); assertRows(execute("SELECT * FROM %s"), row(1, set(userType(1), userType(2)))); flush(); assertRows(execute("SELECT * FROM %s WHERE x = 1"), row(1, set(userType(1), userType(2)))); execute("ALTER TYPE " + KEYSPACE + "." + ut1 + " ADD b int"); execute("UPDATE %s SET y = y + ? WHERE x = 1", set(userType(1, 1), userType(1, 2), userType(2, 1))); flush(); assertRows(execute("SELECT * FROM %s WHERE x = 1"), row(1, set(userType(1), userType(1, 1), userType(1, 2), userType(2), userType(2, 1)))); compact(); assertRows(execute("SELECT * FROM %s WHERE x = 1"), row(1, set(userType(1), userType(1, 1), userType(1, 2), userType(2), userType(2, 1)))); } finally { enableCompaction(); } } {code} There is in fact 2 problems: # When the {{sets}} from the 2 versions are merged the {{ColumnDefinition}} being picked up can be the older one. In which case when the tuples are sorted it my lead to an {{IndexOutOfBoundException}}. # During compaction, the old column definition can be the one being kept for the SSTable metadata. If it is the case the SSTable will not be readable any more and will be marked as {{corrupted}}. > Adding a field to an UDT can corrupte the tables using it > --------------------------------------------------------- > > Key: CASSANDRA-13776 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13776 > Project: Cassandra > Issue Type: Bug > Reporter: Benjamin Lerer > Assignee: Benjamin Lerer > Priority: Critical > > Adding a field to an UDT which is used as a {{Set}} element or as a {{Map}} element can corrupt the table. > The problem can be reproduced using the following test case: > {code} > @Test > public void testReadAfterAlteringUserTypeNestedWithinSet() throws Throwable > { > String ut1 = createType("CREATE TYPE %s (a int)"); > String columnType = KEYSPACE + "." + ut1; > try > { > createTable("CREATE TABLE %s (x int PRIMARY KEY, y set>)"); > disableCompaction(); > execute("INSERT INTO %s (x, y) VALUES(1, ?)", set(userType(1), userType(2))); > assertRows(execute("SELECT * FROM %s"), row(1, set(userType(1), userType(2)))); > flush(); > assertRows(execute("SELECT * FROM %s WHERE x = 1"), > row(1, set(userType(1), userType(2)))); > execute("ALTER TYPE " + KEYSPACE + "." + ut1 + " ADD b int"); > execute("UPDATE %s SET y = y + ? WHERE x = 1", > set(userType(1, 1), userType(1, 2), userType(2, 1))); > flush(); > assertRows(execute("SELECT * FROM %s WHERE x = 1"), > row(1, set(userType(1), > userType(1, 1), > userType(1, 2), > userType(2), > userType(2, 1)))); > compact(); > assertRows(execute("SELECT * FROM %s WHERE x = 1"), > row(1, set(userType(1), > userType(1, 1), > userType(1, 2), > userType(2), > userType(2, 1)))); > } > finally > { > enableCompaction(); > } > } > {code} > There are in fact 2 problems: > # When the {{sets}} from the 2 versions are merged the {{ColumnDefinition}} being picked up can be the older one. In which case when the tuples are sorted it my lead to an {{IndexOutOfBoundsException}}. > # During compaction, the old column definition can be the one being kept for the SSTable metadata. If it is the case the SSTable will not be readable any more and will be marked as {{corrupted}}. > If one of the tables using the type has a Materialized View attached to it, the MV updates can also fail with {{IndexOutOfBoundsException}}. > This problem can be reproduced using the following test: > {code} > @Test > public void testAlteringUserTypeNestedWithinSetWithView() throws Throwable > { > String columnType = typeWithKs(createType("CREATE TYPE %s (a int)")); > createTable("CREATE TABLE %s (pk int, c int, v int, s set>, PRIMARY KEY (pk, c))"); > execute("CREATE MATERIALIZED VIEW " + keyspace() + ".view1 AS SELECT c, pk, v FROM %s WHERE pk IS NOT NULL AND c IS NOT NULL AND v IS NOT NULL PRIMARY KEY (c, pk)"); > execute("INSERT INTO %s (pk, c, v, s) VALUES(?, ?, ?, ?)", 1, 1, 1, set(userType(1), userType(2))); > flush(); > execute("ALTER TYPE " + columnType + " ADD b int"); > execute("UPDATE %s SET s = s + ?, v = ? WHERE pk = ? AND c = ?", > set(userType(1, 1), userType(1, 2), userType(2, 1)), 2, 1, 1); > assertRows(execute("SELECT * FROM %s WHERE pk = ? AND c = ?", 1, 1), > row(1, 1, 2, set(userType(1), > userType(1, 1), > userType(1, 2), > userType(2), > userType(2, 1)))); > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org For additional commands, e-mail: commits-help@cassandra.apache.org