cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "J.P. Eiti Kimura (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10233) IndexOutOfBoundsException in HintedHandOffManager
Date Fri, 02 Oct 2015 12:47:27 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941087#comment-14941087
] 

J.P. Eiti Kimura commented on CASSANDRA-10233:
----------------------------------------------

Good set of info [~fhsgoncalves], thanks. 
[~nutbunnies] it is working now. My main concern is related to consistence, if it happens
again (another issue) or with other people affected by problem may cause cluster problems
if hints were not replied correctly.

The change in the patch will continue show the exception in system.log but will not stop the
flow, we'll skip the inconsistent line and move to the next, continue to schedule the hints
and hand off process without affecting the entire cluster consistency. 
I think having a bunch of nodes in the cluster with problems sending hints a big issue. The
patch is not to fix the hints but will prevent things get any worse.

Do you consider the possibility to apply this patch in the near future? 



> IndexOutOfBoundsException in HintedHandOffManager
> -------------------------------------------------
>
>                 Key: CASSANDRA-10233
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10233
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 2.2.0
>            Reporter: Omri Iluz
>            Assignee: Andrew Hust
>         Attachments: cassandra-2.1.8-10233-v2.txt, cassandra-2.1.8-10233.txt
>
>
> After upgrading our cluster to 2.2.0, the following error started showing exectly every
10 minutes on every server in the cluster:
> {noformat}
> INFO  [CompactionExecutor:1381] 2015-08-31 18:31:55,506 CompactionTask.java:142 - Compacting
(8e7e1520-500e-11e5-b1e3-e95897ba4d20) [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-540-big-Data.db:level=0,
]
> INFO  [CompactionExecutor:1381] 2015-08-31 18:31:55,599 CompactionTask.java:224 - Compacted
(8e7e1520-500e-11e5-b1e3-e95897ba4d20) 1 sstables to [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-541-big,]
to level=0.  1,544,495 bytes to 1,544,495 (~100% of original) in 93ms = 15.838121MB/s.  0
total partitions merged to 4.  Partition merge counts were {1:4, }
> ERROR [HintedHandoff:1] 2015-08-31 18:31:55,600 CassandraDaemon.java:182 - Exception
in thread Thread[HintedHandoff:1,1,main]
> java.lang.IndexOutOfBoundsException: null
> 	at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_79]
> 	at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:410) ~[na:1.7.0_79]
> 	at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106) ~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:515)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
~[apache-cassandra-2.2.0.jar:2.2.0]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79]
> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_79]
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_79]
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_79]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message