incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: "Final buffer length 4690 to accomodate data size of 2347 for RowMutation" error caused node death
Date Mon, 20 Feb 2012 20:10:50 GMT
Can you provide the definition for the PlayerCity CF, the client you are using and any other
info about the mutation ? 

It's a strange looking mutation. There are a lot of super columns are not marked for delete
and do not have nay columns in them. 
 
The error occurred because it took 3 bytes more than expected to serialise the mutation. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/02/2012, at 12:43 AM, Thomas van Neerijnen wrote:

> Hi all
> 
> I am running the Apache packaged Cassandra 1.0.7 on Ubuntu 11.10.
> It has been running fine for over a month however I encountered the below error yesterday
which almost immediately resulted in heap usage rising quickly to almost 100% and client requests
timing out on the affected node.
> I gave up waiting for the init script to stop Cassandra and killed it myself after about
3 minutes, restarted it and it has been fine since. Anyone seen this before?
> 
> Here is the error in the output.log:
> 
> ERROR 10:51:44,282 Fatal exception in thread Thread[COMMIT-LOG-WRITER,5,main]
> java.lang.AssertionError: Final buffer length 4690 to accomodate data size of 2347 (predicted
2344) for RowMutation(keyspace='Player', key='36336138643338652d366162302d343334392d383466302d356166643863353133356465',
modifications=[ColumnFamily(PlayerCity [SuperColumn(owneditem_1019 []),SuperColumn(owneditem_1024
[]),SuperColumn(owneditem_1026 []),SuperColumn(owneditem_1074 []),SuperColumn(owneditem_1077
[]),SuperColumn(owneditem_1084 []),SuperColumn(owneditem_1094 []),SuperColumn(owneditem_1130
[]),SuperColumn(owneditem_1136 []),SuperColumn(owneditem_1141 []),SuperColumn(owneditem_1142
[]),SuperColumn(owneditem_1145 []),SuperColumn(owneditem_1218 [636f6e6e6563746564:false:5@1329648704269002,63757272656e744865616c7468:false:3@1329648704269006,656e64436f6e737472756374696f6e54696d65:false:13@1329648704269007,6964:false:4@1329648704269000,6974656d4964:false:15@1329648704269001,6c61737444657374726f79656454696d65:false:1@1329648704269008,6c61737454696d65436f6c6c6563746564:false:13@1329648704269005,736b696e4964:false:7@1329648704269009,78:false:4@1329648704269003,79:false:3@1329648704269004,]),SuperColumn(owneditem_133
[]),SuperColumn(owneditem_134 []),SuperColumn(owneditem_135 []),SuperColumn(owneditem_141
[]),SuperColumn(owneditem_147 []),SuperColumn(owneditem_154 []),SuperColumn(owneditem_159
[]),SuperColumn(owneditem_171 []),SuperColumn(owneditem_253 []),SuperColumn(owneditem_422
[]),SuperColumn(owneditem_438 []),SuperColumn(owneditem_515 []),SuperColumn(owneditem_521
[]),SuperColumn(owneditem_523 []),SuperColumn(owneditem_525 []),SuperColumn(owneditem_562
[]),SuperColumn(owneditem_61 []),SuperColumn(owneditem_634 []),SuperColumn(owneditem_636 []),SuperColumn(owneditem_71
[]),SuperColumn(owneditem_712 []),SuperColumn(owneditem_720 []),SuperColumn(owneditem_728
[]),SuperColumn(owneditem_787 []),SuperColumn(owneditem_797 []),SuperColumn(owneditem_798
[]),SuperColumn(owneditem_838 []),SuperColumn(owneditem_842 []),SuperColumn(owneditem_847
[]),SuperColumn(owneditem_849 []),SuperColumn(owneditem_851 []),SuperColumn(owneditem_852
[]),SuperColumn(owneditem_853 []),SuperColumn(owneditem_854 []),SuperColumn(owneditem_857
[]),SuperColumn(owneditem_858 []),SuperColumn(owneditem_874 []),SuperColumn(owneditem_884
[]),SuperColumn(owneditem_886 []),SuperColumn(owneditem_908 []),SuperColumn(owneditem_91 []),SuperColumn(owneditem_911
[]),SuperColumn(owneditem_930 []),SuperColumn(owneditem_934 []),SuperColumn(owneditem_937
[]),SuperColumn(owneditem_944 []),SuperColumn(owneditem_945 []),SuperColumn(owneditem_962
[]),SuperColumn(owneditem_963 []),SuperColumn(owneditem_964 []),])])
>         at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682)
>         at org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:279)
>         at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:122)
>         at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:599)
>         at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at java.lang.Thread.run(Thread.java:662)
>  WARN 10:51:54,302 Heap is 0.764063958911146 full.  You may need to reduce memtable and/or
cache sizes.  Cassandra will now flush up to the two largest memtables to free up memory.
 Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra
to do this automatically
>  WARN 10:51:54,303 Flushing CFS(Keyspace='Player', ColumnFamily='PlayerDetail') to relieve
memory pressure
>  INFO 11:00:41,162 Started hinted handoff for token: 121529416757478022665490931225631504090
with IP: /10.16.96.212
>  INFO 11:00:41,163 Finished hinted handoff of 0 rows to endpoint /10.16.96.212
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor192]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor165]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor202]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor232]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor146]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor181]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor190]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor214]
> ...
> [Unloading class sun.reflect.GeneratedConstructorAccessor19]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor209]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor227]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor205]
> [Unloading class sun.reflect.GeneratedMethodAccessor202]
> 
> The "Unloading" messages continued, with occasional mentions of hinted handoffs in between,
until I killed Cassandra.


Mime
View raw message