cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10233) IndexOutOfBoundsException in HintedHandOffManager
Date Sat, 24 Oct 2015 00:25:27 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972173#comment-14972173
] 

Paulo Motta commented on CASSANDRA-10233:
-----------------------------------------

[~fhsgoncalves] [~eitikimura]

bq. the problem started when we added 7 new nodes to the cluster a week ago

Do you remember if there was any failed bootstrap when you added these nodes? For example,
you started bootstrapping, streams hanged, you wiped the node and restarted the process again?
Or did all bootstraps succeed in the first time? (I'm investigating the root cause of the
issue on CASSANDRA-10485)

> IndexOutOfBoundsException in HintedHandOffManager
> -------------------------------------------------
>
>                 Key: CASSANDRA-10233
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10233
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 2.2.0
>            Reporter: Omri Iluz
>            Assignee: J.P. Eiti Kimura
>            Priority: Minor
>             Fix For: 2.2.4, 2.1.12
>
>         Attachments: cassandra-2.0-10233.txt, cassandra-2.1-10233-v5.txt, cassandra-2.1.8-10233-v3.txt,
cassandra-2.1.8-10233-v4.txt, cassandra-2.2.1-10233-v2.txt, cassandra-2.2.1-10233.txt
>
>
> After upgrading our cluster to 2.2.0, the following error started showing exectly every
10 minutes on every server in the cluster:
> {noformat}
> INFO  [CompactionExecutor:1381] 2015-08-31 18:31:55,506 CompactionTask.java:142 - Compacting
(8e7e1520-500e-11e5-b1e3-e95897ba4d20) [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-540-big-Data.db:level=0,
]
> INFO  [CompactionExecutor:1381] 2015-08-31 18:31:55,599 CompactionTask.java:224 - Compacted
(8e7e1520-500e-11e5-b1e3-e95897ba4d20) 1 sstables to [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-541-big,]
to level=0.  1,544,495 bytes to 1,544,495 (~100% of original) in 93ms = 15.838121MB/s.  0
total partitions merged to 4.  Partition merge counts were {1:4, }
> ERROR [HintedHandoff:1] 2015-08-31 18:31:55,600 CassandraDaemon.java:182 - Exception
in thread Thread[HintedHandoff:1,1,main]
> java.lang.IndexOutOfBoundsException: null
> 	at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_79]
> 	at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:410) ~[na:1.7.0_79]
> 	at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106) ~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:515)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_79]
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_79]
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_79]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message