cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Theo Chu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-13339) java.nio.BufferOverflowException: null
Date Fri, 01 Sep 2017 03:04:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149968#comment-16149968
] 

Theo Chu edited comment on CASSANDRA-13339 at 9/1/17 3:03 AM:
--------------------------------------------------------------

I saw the same exception as well happening a couple of times (~30 times over 12 hours) on
two nodes in one of the datacenters. We are running a 4 datacenter cluster with 4 nodes per
datacenter. 

This particular datacenter is write heavy and handles a mix of selects, updates, inserts and
deletes, in some cases with unlogged batches on the same partition key. No use of logged batches
or "IF NOT EXISTS" clauses. Cassandra version 3.0.14.



was (Author: theochu):
I saw the same exception as well happening a couple of times (~30 times over 12 hours) on
two nodes. We are running a 4 datacenter cluster with 4 nodes per datacenter. 

This particular datacenter is write heavy and handles a mix of selects, updates, inserts and
deletes, in some cases with unlogged batches on the same partition key. No use of logged batches
or "IF NOT EXISTS" clauses. Cassandra version 3.0.14.


> java.nio.BufferOverflowException: null
> --------------------------------------
>
>                 Key: CASSANDRA-13339
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13339
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Chris Richards
>
> I'm seeing the following exception running Cassandra 3.9 (with Netty updated to 4.1.8.Final)
running on a 2 node cluster.  It would have been processing around 50 queries/second at the
time (mixture of inserts/updates/selects/deletes) : there's a collection of tables (some with
counters some without) and a single materialized view.
> {code}
> ERROR [MutationStage-4] 2017-03-15 22:50:33,052 StorageProxy.java:1353 - Failed to apply
mutation locally : {}
> java.nio.BufferOverflowException: null
> 	at org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:262)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.EncodingStats$Serializer.serialize(EncodingStats.java:233)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:380)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:122)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:790)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:393)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:279) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:227) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:241) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1347) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2539)
[apache-cassandra-3.9.jar:3.9]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {code}
> and then again shortly afterwards
> {code}
> ERROR [MutationStage-3] 2017-03-15 23:27:36,198 StorageProxy.java:1353 - Failed to apply
mutation locally : {}
> java.nio.BufferOverflowException: null
> 	at org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:262)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.EncodingStats$Serializer.serialize(EncodingStats.java:233)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:380)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:122)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:790)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:393)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:279) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:227) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:241) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1347) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2539)
[apache-cassandra-3.9.jar:3.9]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message