cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10579) IndexOutOfBoundsException during memtable flushing at startup (with offheap_objects)
Date Tue, 27 Oct 2015 17:38:27 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14976810#comment-14976810
] 

Benedict commented on CASSANDRA-10579:
--------------------------------------

I based the patch on 2.1.10, so you should be good to roll out the {{-fix}} patch across your
cluster, as it has no other changes. If you'd prefer to wait until formal review occurs, it
will be exactly the same as deploying 2.1.11 (when it materializes). However the patch will
only be made officially available in 2.1.11, as we do not re-release.

Taking another look, it seems that without assertions the possibility for undefined behaviour
(potentially resulting in corruption) is quite high, as negative lengths can occur in the
data we serialize (at minimum).

I've pushed an update that includes unit tests to cover this issue.

> IndexOutOfBoundsException during memtable flushing at startup (with offheap_objects)
> ------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-10579
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10579
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: 2.1.10 on linux
>            Reporter: Jeff Griffith
>            Assignee: Benedict
>             Fix For: 2.1.x
>
>
> Sometimes we have problems at startup where memtable flushes with an index out of bounds
exception as seen below. Cassandra is then dead in the water until we track down the corresponding
commit log via the segment ID and remove it:
> {code}
> INFO  [main] 2015-10-23 14:43:36,440 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log
> INFO  [main] 2015-10-23 14:43:36,440 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log
(CL version 4, messaging version 8)
> INFO  [main] 2015-10-23 14:43:36,594 CommitLogReplayer.java:478 - Finished reading /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log
> INFO  [main] 2015-10-23 14:43:36,594 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log
> INFO  [main] 2015-10-23 14:43:36,595 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log
(CL version 4, messaging version 8)
> INFO  [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:478 - Finished reading /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log
> INFO  [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832694.log
> INFO  [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832694.log
(CL version 4, messaging version 8)
> WARN  [SharedPool-Worker-5] 2015-10-23 14:43:36,747 AbstractTracingAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-5,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 6
>         at org.apache.cassandra.db.AbstractNativeCell.nametype(AbstractNativeCell.java:204)
~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.db.AbstractNativeCell.isStatic(AbstractNativeCell.java:199)
~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.db.composites.AbstractCType.compare(AbstractCType.java:166)
~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.db.composites.AbstractCellNameType$1.compare(AbstractCellNameType.java:61)
~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.db.composites.AbstractCellNameType$1.compare(AbstractCellNameType.java:58)
~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.utils.btree.BTree.find(BTree.java:277) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:154)
~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.utils.btree.Builder.update(Builder.java:74) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.utils.btree.BTree.update(BTree.java:186) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:225)
~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.db.Memtable.put(Memtable.java:210) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1225)
~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:455)
~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_31]
>         at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message