cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11050) SSTables not loaded after dropping column
Date Fri, 22 Jan 2016 17:27:39 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112735#comment-15112735
] 

Sylvain Lebresne commented on CASSANDRA-11050:
----------------------------------------------

bq.  do you have any concerns about the schema hash changing ?

Actually I have (but I was too fast and committed before realizing this was indeed a problem
which is why it's been committed and then reverted).

Unless I'm missing something in our code, the change to the hash means that during an upgrade
to a version with this patch, old and new nodes will never agree on the schema version (only
if they have dropped columns but still) and will thus continuously exchange schema. That's
not ideal. But we still need to fix that bug.

Right off the bat I can see the following options:
# we just accept that there will be disagreement during upgrade. Having node exchange schema
for no reason is not great but I believe we'll do that at most once every minute which hopefully
shouldn't be hugely noticeable (of course, assuming you have a reasonable amount of keyspaces/tables).
Though it's worth noting this will disturb clients if they wait for schema agreement. And
3.x is very young so we could count that as one of the bump in the road. It's scary though.
# we special case the schema has code to ignore that table (but include it for anything else)
and this until 4.0 (where, since it's a major version, there won't be schema hash comparison
involved). The downside being that we could get into a situation where a node doesn't know
about some dropped column and it doesn't get fixed. On the one side it's not ideal (to leave
a known bug in), but on the other side you'd have to be unlucky to have that be a real problem,
and if you do get there, there is an easy workaround as you can reset your schema version
and thus force nodes to re-synchronize on the schema.

Anway, no option is perfect but the 2nd one feels a tad less scary. Any other opinion ([~iamaleksey]
in particular since you've been deep in schema code lately).

As a side note, we should really write a dtest for this issue before committing ([~amorton]
would you be up for that?).

> SSTables not loaded after dropping column
> -----------------------------------------
>
>                 Key: CASSANDRA-11050
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11050
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Distributed Metadata
>            Reporter: amorton
>            Assignee: amorton
>         Attachments: 11050-3.0.patch
>
>
> The {{system_schema.dropped_tables}} table is not flushed when schema is updated, this
can result in SSTables not being loaded at startup and failure to start if the commit log
contains mutations with the column. 
> Reproduce on cassandra-3.0 branch by starting a node and running following in cqlsh:
> {code}
> create keyspace dev WITH replication = {'class':'SimpleStrategy', 'replication_factor':1};
> use dev;
> create table foo (
>      foo text primary key,
>      bar text,
>      baz text
> );
> insert into foo (foo, bar, baz) values ('foo','this is bar', 'this is baz');
> alter table foo 
> drop baz;
> {code}
> Stop the node and restart, the following errors are raised and the node does not start:
> {code}
> ERROR 16:38:19 Exception in thread Thread[SSTableBatchOpen:1,5,main]
> java.lang.RuntimeException: Unknown column baz during deserialization
> 	at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:485)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
~[main/:na]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> ...
> ERROR 16:38:19 Exiting due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Unexpected
error deserializing mutation; saved to /var/folders/r2/rkv1jz3j0j74r9s1zm5xx9wc0000gn/T/mutation5408885979635225676dat.
 This may be caused by replaying a mutation against a table with the same name but incompatible
schema.  Exception follows: java.lang.RuntimeException: Unknown column baz during deserialization
> 	at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
[main/:na]
> 	at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
[main/:na]
> 	at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
[main/:na]
> 	at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404)
[main/:na]
> 	at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151)
[main/:na]
> 	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) [main/:na]
> 	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) [main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [main/:na]
> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679) [main/:na]
> {code}
> {{dropped_columns}} is not in the list of tables to flush, {{SchemaKeyspace.ALL}}. 
> It's a simple patch to add it, attached. The fix will need to go to 3.0, 3.1 and trunk
AFAIK
> however this will change the way the schema hash is calculated in {{SchemaKeyspace.calculateSchemaDigest()}}
It looks like this would cause the nodes to announce a new version of the schema on (first)
restart. 
> I  currently donit understand all the implications of changing the schema hash, thoughts
? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message