Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 125A318FB1 for ; Tue, 27 Oct 2015 17:38:28 +0000 (UTC) Received: (qmail 61309 invoked by uid 500); 27 Oct 2015 17:38:27 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 61264 invoked by uid 500); 27 Oct 2015 17:38:27 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 61240 invoked by uid 99); 27 Oct 2015 17:38:27 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Oct 2015 17:38:27 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id B97A32C1F58 for ; Tue, 27 Oct 2015 17:38:27 +0000 (UTC) Date: Tue, 27 Oct 2015 17:38:27 +0000 (UTC) From: "Benedict (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-10579) IndexOutOfBoundsException during memtable flushing at startup (with offheap_objects) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14976810#comment-14976810 ] Benedict commented on CASSANDRA-10579: -------------------------------------- I based the patch on 2.1.10, so you should be good to roll out the {{-fix}} patch across your cluster, as it has no other changes. If you'd prefer to wait until formal review occurs, it will be exactly the same as deploying 2.1.11 (when it materializes). However the patch will only be made officially available in 2.1.11, as we do not re-release. Taking another look, it seems that without assertions the possibility for undefined behaviour (potentially resulting in corruption) is quite high, as negative lengths can occur in the data we serialize (at minimum). I've pushed an update that includes unit tests to cover this issue. > IndexOutOfBoundsException during memtable flushing at startup (with offheap_objects) > ------------------------------------------------------------------------------------ > > Key: CASSANDRA-10579 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10579 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.1.10 on linux > Reporter: Jeff Griffith > Assignee: Benedict > Fix For: 2.1.x > > > Sometimes we have problems at startup where memtable flushes with an index out of bounds exception as seen below. Cassandra is then dead in the water until we track down the corresponding commit log via the segment ID and remove it: > {code} > INFO [main] 2015-10-23 14:43:36,440 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log > INFO [main] 2015-10-23 14:43:36,440 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log (CL version 4, messaging version 8) > INFO [main] 2015-10-23 14:43:36,594 CommitLogReplayer.java:478 - Finished reading /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log > INFO [main] 2015-10-23 14:43:36,594 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log > INFO [main] 2015-10-23 14:43:36,595 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log (CL version 4, messaging version 8) > INFO [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:478 - Finished reading /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log > INFO [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832694.log > INFO [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832694.log (CL version 4, messaging version 8) > WARN [SharedPool-Worker-5] 2015-10-23 14:43:36,747 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-5,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 6 > at org.apache.cassandra.db.AbstractNativeCell.nametype(AbstractNativeCell.java:204) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.AbstractNativeCell.isStatic(AbstractNativeCell.java:199) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.composites.AbstractCType.compare(AbstractCType.java:166) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.composites.AbstractCellNameType$1.compare(AbstractCellNameType.java:61) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.composites.AbstractCellNameType$1.compare(AbstractCellNameType.java:58) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.utils.btree.BTree.find(BTree.java:277) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:154) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.utils.btree.Builder.update(Builder.java:74) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.utils.btree.BTree.update(BTree.java:186) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:225) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.Memtable.put(Memtable.java:210) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1225) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:455) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_31] > at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)