cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Eriksson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13619) java.nio.BufferOverflowException: null while flushing hints
Date Thu, 07 Sep 2017 12:51:01 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156907#comment-16156907
] 

Marcus Eriksson commented on CASSANDRA-13619:
---------------------------------------------

[~milansm] could you post your schema and the modification statements you are running?

> java.nio.BufferOverflowException: null while flushing hints
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-13619
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13619
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Coordination, Core
>            Reporter: Milan Milosevic
>
> I'm seeing the following exception running Cassandra 3.0.11 on 21 node cluster in two
AWS regions when half of the nodes in one region go down, and the load is high on the rest
of the nodes:
> {code}
> WARN  [SharedPool-Worker-10] 2017-06-14 12:57:15,017 AbstractLocalAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-10,5,main]: {}
> java.lang.RuntimeException: java.nio.BufferOverflowException
>         at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2549)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0-zing_17.03.1.0]
>         at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.11.jar:3.0.11]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0-zing_17.03.1.0]
> Caused by: java.nio.BufferOverflowException: null
>         at org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:195)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:258)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.db.Columns$Serializer.serialize(Columns.java:405) ~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:407)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:120)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:625)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:305)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.hints.Hint$Serializer.serialize(Hint.java:141) ~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.hints.HintsBuffer$Allocation.write(HintsBuffer.java:251)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.hints.HintsBuffer$Allocation.write(HintsBuffer.java:230)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.hints.HintsBufferPool.write(HintsBufferPool.java:61)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.hints.HintsService.write(HintsService.java:154) ~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.service.StorageProxy$11.runMayThrow(StorageProxy.java:2627)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2545)
~[apache-cassandra-3.0.11.jar:3.0.11]
>         ... 5 common frames omitted
> {code}
> Relevant configurations from cassandra.yaml:
> {code}
> -cassandra_hinted_handoff_throttle_in_kb: 1024
>  cassandra_max_hints_delivery_threads: 4
> -cassandra_hints_flush_period_in_ms: 10000
> -cassandra_max_hints_file_size_in_mb: 512
> {code}
> When I reduce -cassandra_hints_flush_period_in_ms: 10000 to 5000, the number of exceptions
lowers significantly, but they are still present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message