cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean Tremblay <jean.tremb...@zen-innovations.com>
Subject Cassandra 3.1.1 with respect to HeapSpace
Date Thu, 14 Jan 2016 16:56:10 GMT
Hi,

I have a small Cassandra Cluster with 5 nodes, having 16MB of RAM.
I use Cassandra 3.1.1.
I use the following setup for the memory:
  MAX_HEAP_SIZE="6G"
HEAP_NEWSIZE="496M"

I have been loading a lot of data in this cluster over the last 24 hours. The system behaved
I think very nicely. It was loading very fast, and giving excellent read time. There was no
error messages until this one:


ERROR [SharedPool-Worker-35] 2016-01-14 17:05:23,602 JVMStabilityInspector.java:139 - JVM
state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_65]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_65]
at org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126) ~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) ~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:297)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374) ~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.rows.BufferCell$Serializer.serialize(BufferCell.java:263) ~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:183)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:108)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:96)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:132)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:298)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:136) ~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:128)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) ~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) ~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-3.1.1.jar:3.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_65]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.1.1.jar:3.1.1]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

4 nodes out of 5 crashed with this error message. Now when I want to restart the first node
I have the following error;

ERROR [main] 2016-01-14 17:15:59,617 JVMStabilityInspector.java:81 - Exiting due to error
while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Unexpected error
deserializing mutation; saved to /tmp/mutation7465380878750576105dat.  This may be caused
by replaying a mutation against a table with the same name but incompatible schema.  Exception
follows: org.apache.cassandra.serializers.MarshalException: Not enough bytes to read a map
at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404)
[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151)
[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) [apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) [apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:549) [apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:677) [apache-cassandra-3.1.1.jar:3.1.1]

I can no longer start my nodes.

How can I restart my cluster?
Is this problem known?
Is there a better Cassandra 3 version which would behave better with respect to this problem?
Would there be a better memory configuration to select for my nodes? Currently I use MAX_HEAP_SIZE="6G"
HEAP_NEWSIZE=“496M” for a 16M RAM node.


Thank you very much for your advice.

Kind regards

Jean
Mime
View raw message