cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: "Final buffer length 4690 to accomodate data size of 2347 for RowMutation" error caused node death
Date Fri, 24 Feb 2012 22:07:20 GMT
I've filed https://issues.apache.org/jira/browse/CASSANDRA-3957 as a
bug.  Any further light you can shed here would be useful.  (Is row
cache enabled?  Is JNA installed?)

On Mon, Feb 20, 2012 at 5:43 AM, Thomas van Neerijnen
<tom@bossastudios.com> wrote:
> Hi all
>
> I am running the Apache packaged Cassandra 1.0.7 on Ubuntu 11.10.
> It has been running fine for over a month however I encountered the below
> error yesterday which almost immediately resulted in heap usage rising
> quickly to almost 100% and client requests timing out on the affected node.
> I gave up waiting for the init script to stop Cassandra and killed it myself
> after about 3 minutes, restarted it and it has been fine since. Anyone seen
> this before?
>
> Here is the error in the output.log:
>
> ERROR 10:51:44,282 Fatal exception in thread
> Thread[COMMIT-LOG-WRITER,5,main]
> java.lang.AssertionError: Final buffer length 4690 to accomodate data size
> of 2347 (predicted 2344) for RowMutation(keyspace='Player',
> key='36336138643338652d366162302d343334392d383466302d356166643863353133356465',
> modifications=[ColumnFamily(PlayerCity [SuperColumn(owneditem_1019
> []),SuperColumn(owneditem_1024 []),SuperColumn(owneditem_1026
> []),SuperColumn(owneditem_1074 []),SuperColumn(owneditem_1077
> []),SuperColumn(owneditem_1084 []),SuperColumn(owneditem_1094
> []),SuperColumn(owneditem_1130 []),SuperColumn(owneditem_1136
> []),SuperColumn(owneditem_1141 []),SuperColumn(owneditem_1142
> []),SuperColumn(owneditem_1145 []),SuperColumn(owneditem_1218
> [636f6e6e6563746564:false:5@1329648704269002,63757272656e744865616c7468:false:3@1329648704269006,656e64436f6e737472756374696f6e54696d65:false:13@1329648704269007,6964:false:4@1329648704269000,6974656d4964:false:15@1329648704269001,6c61737444657374726f79656454696d65:false:1@1329648704269008,6c61737454696d65436f6c6c6563746564:false:13@1329648704269005,736b696e4964:false:7@1329648704269009,78:false:4@1329648704269003,79:false:3@1329648704269004,]),SuperColumn(owneditem_133
> []),SuperColumn(owneditem_134 []),SuperColumn(owneditem_135
> []),SuperColumn(owneditem_141 []),SuperColumn(owneditem_147
> []),SuperColumn(owneditem_154 []),SuperColumn(owneditem_159
> []),SuperColumn(owneditem_171 []),SuperColumn(owneditem_253
> []),SuperColumn(owneditem_422 []),SuperColumn(owneditem_438
> []),SuperColumn(owneditem_515 []),SuperColumn(owneditem_521
> []),SuperColumn(owneditem_523 []),SuperColumn(owneditem_525
> []),SuperColumn(owneditem_562 []),SuperColumn(owneditem_61
> []),SuperColumn(owneditem_634 []),SuperColumn(owneditem_636
> []),SuperColumn(owneditem_71 []),SuperColumn(owneditem_712
> []),SuperColumn(owneditem_720 []),SuperColumn(owneditem_728
> []),SuperColumn(owneditem_787 []),SuperColumn(owneditem_797
> []),SuperColumn(owneditem_798 []),SuperColumn(owneditem_838
> []),SuperColumn(owneditem_842 []),SuperColumn(owneditem_847
> []),SuperColumn(owneditem_849 []),SuperColumn(owneditem_851
> []),SuperColumn(owneditem_852 []),SuperColumn(owneditem_853
> []),SuperColumn(owneditem_854 []),SuperColumn(owneditem_857
> []),SuperColumn(owneditem_858 []),SuperColumn(owneditem_874
> []),SuperColumn(owneditem_884 []),SuperColumn(owneditem_886
> []),SuperColumn(owneditem_908 []),SuperColumn(owneditem_91
> []),SuperColumn(owneditem_911 []),SuperColumn(owneditem_930
> []),SuperColumn(owneditem_934 []),SuperColumn(owneditem_937
> []),SuperColumn(owneditem_944 []),SuperColumn(owneditem_945
> []),SuperColumn(owneditem_962 []),SuperColumn(owneditem_963
> []),SuperColumn(owneditem_964 []),])])
>         at
> org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682)
>         at
> org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:279)
>         at
> org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:122)
>         at
> org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:599)
>         at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at java.lang.Thread.run(Thread.java:662)
>  WARN 10:51:54,302 Heap is 0.764063958911146 full.  You may need to reduce
> memtable and/or cache sizes.  Cassandra will now flush up to the two largest
> memtables to free up memory.  Adjust flush_largest_memtables_at threshold in
> cassandra.yaml if you don't want Cassandra to do this automatically
>  WARN 10:51:54,303 Flushing CFS(Keyspace='Player',
> ColumnFamily='PlayerDetail') to relieve memory pressure
>  INFO 11:00:41,162 Started hinted handoff for token:
> 121529416757478022665490931225631504090 with IP: /10.16.96.212
>  INFO 11:00:41,163 Finished hinted handoff of 0 rows to endpoint
> /10.16.96.212
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor192]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor165]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor202]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor232]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor146]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor181]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor190]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor214]
> ...
> [Unloading class sun.reflect.GeneratedConstructorAccessor19]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor209]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor227]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor205]
> [Unloading class sun.reflect.GeneratedMethodAccessor202]
>
> The "Unloading" messages continued, with occasional mentions of hinted
> handoffs in between, until I killed Cassandra.



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message