cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyler Hobbs <ty...@datastax.com>
Subject Re: Cassandra 3.1.1 with respect to HeapSpace
Date Thu, 14 Jan 2016 17:18:57 GMT
I don't think that's a known issue.  Can you open a ticket at
https://issues.apache.org/jira/browse/CASSANDRA and attach your schema
along with the commitlog files and the mutation that was saved to /tmp?

On Thu, Jan 14, 2016 at 10:56 AM, Jean Tremblay <
jean.tremblay@zen-innovations.com> wrote:

> Hi,
>
> I have a small Cassandra Cluster with 5 nodes, having 16MB of RAM.
> I use Cassandra 3.1.1.
> I use the following setup for the memory:
>   MAX_HEAP_SIZE="6G"
> HEAP_NEWSIZE="496M"
>
> I have been loading a lot of data in this cluster over the last 24 hours.
> The system behaved I think very nicely. It was loading very fast, and
> giving excellent read time. There was no error messages until this one:
>
>
> ERROR [SharedPool-Worker-35] 2016-01-14 17:05:23,602
> JVMStabilityInspector.java:139 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_65]
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_65]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:297)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.rows.BufferCell$Serializer.serialize(BufferCell.java:263)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:183)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:108)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:96)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:132)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:298)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:136)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:128)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_65]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.1.1.jar:3.1.1]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.1.1.jar:3.1.1]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
>
> 4 nodes out of 5 crashed with this error message. Now when I want to
> restart the first node I have the following error;
>
> ERROR [main] 2016-01-14 17:15:59,617 JVMStabilityInspector.java:81 -
> Exiting due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /tmp/mutation7465380878750576105dat.  This may be caused by replaying a
> mutation against a table with the same name but incompatible schema.
> Exception follows: org.apache.cassandra.serializers.MarshalException: Not
> enough bytes to read a map
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
> [apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
> [apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
> [apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404)
> [apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151)
> [apache-cassandra-3.1.1.jar:3.1.1]
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [apache-cassandra-3.1.1.jar:3.1.1]
> at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283)
> [apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:549)
> [apache-cassandra-3.1.1.jar:3.1.1]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:677)
> [apache-cassandra-3.1.1.jar:3.1.1]
>
> I can no longer start my nodes.
>
> How can I restart my cluster?
> Is this problem known?
> Is there a better Cassandra 3 version which would behave better with
> respect to this problem?
> Would there be a better memory configuration to select for my nodes?
> Currently I use MAX_HEAP_SIZE="6G" HEAP_NEWSIZE=“496M” for a 16M RAM node.
>
>
> Thank you very much for your advice.
>
> Kind regards
>
> Jean
>



-- 
Tyler Hobbs
DataStax <http://datastax.com/>

Mime
View raw message