cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Richards (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13339) java.nio.BufferOverflowException: null
Date Thu, 23 Mar 2017 11:44:41 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15938151#comment-15938151
] 

Chris Richards commented on CASSANDRA-13339:
--------------------------------------------

I added some debug to see what was causing this : the size of the serialized mutation is changing
between when it was originally calculated and the point that the write occurs.  This seems
to occur at any location within the CommitLog file (not just near the end).

ERROR [MutationStage-1] 2017-03-23 08:42:15,184 CommitLog.java:301 - Caught buffer overflow
exception
ERROR [MutationStage-1] 2017-03-23 08:42:15,184 CommitLog.java:302 - totalSize 73, size 61
ERROR [MutationStage-1] 2017-03-23 08:42:15,184 CommitLog.java:303 - buffer: position 22737351,
limit 22737351, capacity 33554432
ERROR [MutationStage-1] 2017-03-23 08:42:15,184 CommitLog.java:306 - recomputed size 106

where recomputed size is the value of
   Mutation.serializer.serializedSize(mutation, MessagingService.current_version);
after the exception has been thrown and caught.

I assume therefore that the mutation is changing between when the serialized size was calculated
and when it serialized - I have a new build that will try and show more information about
the mutation when this occurs to see if this sheds any light on what is happening.

> java.nio.BufferOverflowException: null
> --------------------------------------
>
>                 Key: CASSANDRA-13339
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13339
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Chris Richards
>
> I'm seeing the following exception running Cassandra 3.9 (with Netty updated to 4.1.8.Final)
running on a 2 node cluster.  It would have been processing around 50 queries/second at the
time (mixture of inserts/updates/selects/deletes) : there's a collection of tables (some with
counters some without) and a single materialized view.
> ERROR [MutationStage-4] 2017-03-15 22:50:33,052 StorageProxy.java:1353 - Failed to apply
mutation locally : {}
> java.nio.BufferOverflowException: null
> 	at org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:262)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.EncodingStats$Serializer.serialize(EncodingStats.java:233)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:380)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:122)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:790)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:393)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:279) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:227) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:241) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1347) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2539)
[apache-cassandra-3.9.jar:3.9]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> and then again shortly afterwards
> ERROR [MutationStage-3] 2017-03-15 23:27:36,198 StorageProxy.java:1353 - Failed to apply
mutation locally : {}
> java.nio.BufferOverflowException: null
> 	at org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:262)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.EncodingStats$Serializer.serialize(EncodingStats.java:233)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:380)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:122)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:790)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:393)
~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:279) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:227) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:241) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1347) ~[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2539)
[apache-cassandra-3.9.jar:3.9]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
> 	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message