Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3DA3418B40 for ; Thu, 1 Oct 2015 22:30:27 +0000 (UTC) Received: (qmail 96890 invoked by uid 500); 1 Oct 2015 22:30:27 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 96853 invoked by uid 500); 1 Oct 2015 22:30:27 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 96838 invoked by uid 99); 1 Oct 2015 22:30:27 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2015 22:30:27 +0000 Date: Thu, 1 Oct 2015 22:30:26 +0000 (UTC) From: "Andrew Hust (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-10233) IndexOutOfBoundsException in HintedHandOffManager MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940540#comment-14940540 ] Andrew Hust commented on CASSANDRA-10233: ----------------------------------------- [~eitikimura] as [~pauloricardomg] mentioned truncating the hints table will get things rolling again, just make sure to repair in near future to prevent possible dataloss or zombie records related to the mutations truncated out of the hints table. [~fhsgoncalves] thanks, that helps a ton. Couple more questions to help reproduce it: - Just want to make sure order of operation is correct, the error showed up when you added additional nodes then persisted after upgrades? - Are all the new nodes in the second rack? - Are you running with vnodes? - Are your keyspaces set with a replication of NetworkTopologyStrategy, SimpleStrategy or ? - In the cassandra.yaml what is the value for: endpoint_snitch > IndexOutOfBoundsException in HintedHandOffManager > ------------------------------------------------- > > Key: CASSANDRA-10233 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10233 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 2.2.0 > Reporter: Omri Iluz > Assignee: Andrew Hust > Attachments: cassandra-2.1.8-10233-v2.txt, cassandra-2.1.8-10233.txt > > > After upgrading our cluster to 2.2.0, the following error started showing exectly every 10 minutes on every server in the cluster: > {noformat} > INFO [CompactionExecutor:1381] 2015-08-31 18:31:55,506 CompactionTask.java:142 - Compacting (8e7e1520-500e-11e5-b1e3-e95897ba4d20) [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-540-big-Data.db:level=0, ] > INFO [CompactionExecutor:1381] 2015-08-31 18:31:55,599 CompactionTask.java:224 - Compacted (8e7e1520-500e-11e5-b1e3-e95897ba4d20) 1 sstables to [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-541-big,] to level=0. 1,544,495 bytes to 1,544,495 (~100% of original) in 93ms = 15.838121MB/s. 0 total partitions merged to 4. Partition merge counts were {1:4, } > ERROR [HintedHandoff:1] 2015-08-31 18:31:55,600 CassandraDaemon.java:182 - Exception in thread Thread[HintedHandoff:1,1,main] > java.lang.IndexOutOfBoundsException: null > at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_79] > at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:410) ~[na:1.7.0_79] > at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106) ~[apache-cassandra-2.2.0.jar:2.2.0] > at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:515) ~[apache-cassandra-2.2.0.jar:2.2.0] > at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88) ~[apache-cassandra-2.2.0.jar:2.2.0] > at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168) ~[apache-cassandra-2.2.0.jar:2.2.0] > at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-2.2.0.jar:2.2.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_79] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_79] > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_79] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)