cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Hust (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10233) IndexOutOfBoundsException in HintedHandOffManager
Date Thu, 01 Oct 2015 22:30:26 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940540#comment-14940540
] 

Andrew Hust commented on CASSANDRA-10233:
-----------------------------------------

[~eitikimura] as [~pauloricardomg] mentioned truncating the hints table will get things rolling
again, just make sure to repair in near future to prevent possible dataloss or zombie records
related to the mutations truncated out of the hints table.

[~fhsgoncalves] thanks, that helps a ton.  Couple more questions to help reproduce it:
- Just want to make sure order of operation is correct, the error showed up when you added
additional nodes then persisted after upgrades?  
- Are all the new nodes in the second rack?  
- Are you running with vnodes?
- Are your keyspaces set with a replication of NetworkTopologyStrategy, SimpleStrategy or
?
- In the cassandra.yaml what is the value for: endpoint_snitch

> IndexOutOfBoundsException in HintedHandOffManager
> -------------------------------------------------
>
>                 Key: CASSANDRA-10233
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10233
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 2.2.0
>            Reporter: Omri Iluz
>            Assignee: Andrew Hust
>         Attachments: cassandra-2.1.8-10233-v2.txt, cassandra-2.1.8-10233.txt
>
>
> After upgrading our cluster to 2.2.0, the following error started showing exectly every
10 minutes on every server in the cluster:
> {noformat}
> INFO  [CompactionExecutor:1381] 2015-08-31 18:31:55,506 CompactionTask.java:142 - Compacting
(8e7e1520-500e-11e5-b1e3-e95897ba4d20) [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-540-big-Data.db:level=0,
]
> INFO  [CompactionExecutor:1381] 2015-08-31 18:31:55,599 CompactionTask.java:224 - Compacted
(8e7e1520-500e-11e5-b1e3-e95897ba4d20) 1 sstables to [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-541-big,]
to level=0.  1,544,495 bytes to 1,544,495 (~100% of original) in 93ms = 15.838121MB/s.  0
total partitions merged to 4.  Partition merge counts were {1:4, }
> ERROR [HintedHandoff:1] 2015-08-31 18:31:55,600 CassandraDaemon.java:182 - Exception
in thread Thread[HintedHandoff:1,1,main]
> java.lang.IndexOutOfBoundsException: null
> 	at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_79]
> 	at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:410) ~[na:1.7.0_79]
> 	at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106) ~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:515)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_79]
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_79]
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_79]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message