incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: "Final buffer length 4690 to accomodate data size of 2347 for RowMutation" error caused node death
Date Wed, 07 Mar 2012 18:23:39 GMT
Thanks, Thomas.

Row cache/CLHCP confirms our suspected culprit.  We've committed a fix
for 1.0.9.

On Wed, Mar 7, 2012 at 11:08 AM, Thomas van Neerijnen
<tom@bossastudios.com> wrote:
> Sorry, for the delay in replying.
>
> I'd like to stress that I've been working on this cluster for many months
> and this was the first and so far last time I got this error so I couldn't
> guess how to duplicate. Sorry I can't be more help.
>
> Anyways, here's the details requested:
> Row caching is enabled, at the time the error occurred using
> ConcurrentLinkedHashCacheProvider.
> It's the Apache packaged version with JNA pulled in as a dependency when I
> installed so yes.
> We're using Hector 1.0.1.
> I'm not sure what was happening at the time the error occured altho the
> empty super columns are expected, assuming my understanding of super columns
> being deleted is correct, which is to say if I delete a super column from a
> row it'll tombstone it and delete the data.
> The schema for PlayerCity is as follows:
>
> create column family PlayerCity
>   with column_type = 'Super'
>   and comparator = 'UTF8Type'
>   and subcomparator = 'BytesType'
>   and default_validation_class = 'BytesType'
>   and key_validation_class = 'BytesType'
>   and rows_cached = 400.0
>   and row_cache_save_period = 0
>   and row_cache_keys_to_save = 2147483647
>   and keys_cached = 200000.0
>   and key_cache_save_period = 14400
>   and read_repair_chance = 1.0
>   and gc_grace = 864000
>   and min_compaction_threshold = 4
>   and max_compaction_threshold = 32
>   and replicate_on_write = true
>   and row_cache_provider = 'ConcurrentLinkedHashCacheProvider'
>   and compaction_strategy =
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy';
>
>
> On Fri, Feb 24, 2012 at 10:07 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> I've filed https://issues.apache.org/jira/browse/CASSANDRA-3957 as a
>> bug.  Any further light you can shed here would be useful.  (Is row
>> cache enabled?  Is JNA installed?)
>>
>> On Mon, Feb 20, 2012 at 5:43 AM, Thomas van Neerijnen
>> <tom@bossastudios.com> wrote:
>> > Hi all
>> >
>> > I am running the Apache packaged Cassandra 1.0.7 on Ubuntu 11.10.
>> > It has been running fine for over a month however I encountered the
>> > below
>> > error yesterday which almost immediately resulted in heap usage rising
>> > quickly to almost 100% and client requests timing out on the affected
>> > node.
>> > I gave up waiting for the init script to stop Cassandra and killed it
>> > myself
>> > after about 3 minutes, restarted it and it has been fine since. Anyone
>> > seen
>> > this before?
>> >
>> > Here is the error in the output.log:
>> >
>> > ERROR 10:51:44,282 Fatal exception in thread
>> > Thread[COMMIT-LOG-WRITER,5,main]
>> > java.lang.AssertionError: Final buffer length 4690 to accomodate data
>> > size
>> > of 2347 (predicted 2344) for RowMutation(keyspace='Player',
>> >
>> > key='36336138643338652d366162302d343334392d383466302d356166643863353133356465',
>> > modifications=[ColumnFamily(PlayerCity [SuperColumn(owneditem_1019
>> > []),SuperColumn(owneditem_1024 []),SuperColumn(owneditem_1026
>> > []),SuperColumn(owneditem_1074 []),SuperColumn(owneditem_1077
>> > []),SuperColumn(owneditem_1084 []),SuperColumn(owneditem_1094
>> > []),SuperColumn(owneditem_1130 []),SuperColumn(owneditem_1136
>> > []),SuperColumn(owneditem_1141 []),SuperColumn(owneditem_1142
>> > []),SuperColumn(owneditem_1145 []),SuperColumn(owneditem_1218
>> >
>> > [636f6e6e6563746564:false:5@1329648704269002,63757272656e744865616c7468:false:3@1329648704269006,656e64436f6e737472756374696f6e54696d65:false:13@1329648704269007,6964:false:4@1329648704269000,6974656d4964:false:15@1329648704269001,6c61737444657374726f79656454696d65:false:1@1329648704269008,6c61737454696d65436f6c6c6563746564:false:13@1329648704269005,736b696e4964:false:7@1329648704269009,78:false:4@1329648704269003,79:false:3@1329648704269004,]),SuperColumn(owneditem_133
>> > []),SuperColumn(owneditem_134 []),SuperColumn(owneditem_135
>> > []),SuperColumn(owneditem_141 []),SuperColumn(owneditem_147
>> > []),SuperColumn(owneditem_154 []),SuperColumn(owneditem_159
>> > []),SuperColumn(owneditem_171 []),SuperColumn(owneditem_253
>> > []),SuperColumn(owneditem_422 []),SuperColumn(owneditem_438
>> > []),SuperColumn(owneditem_515 []),SuperColumn(owneditem_521
>> > []),SuperColumn(owneditem_523 []),SuperColumn(owneditem_525
>> > []),SuperColumn(owneditem_562 []),SuperColumn(owneditem_61
>> > []),SuperColumn(owneditem_634 []),SuperColumn(owneditem_636
>> > []),SuperColumn(owneditem_71 []),SuperColumn(owneditem_712
>> > []),SuperColumn(owneditem_720 []),SuperColumn(owneditem_728
>> > []),SuperColumn(owneditem_787 []),SuperColumn(owneditem_797
>> > []),SuperColumn(owneditem_798 []),SuperColumn(owneditem_838
>> > []),SuperColumn(owneditem_842 []),SuperColumn(owneditem_847
>> > []),SuperColumn(owneditem_849 []),SuperColumn(owneditem_851
>> > []),SuperColumn(owneditem_852 []),SuperColumn(owneditem_853
>> > []),SuperColumn(owneditem_854 []),SuperColumn(owneditem_857
>> > []),SuperColumn(owneditem_858 []),SuperColumn(owneditem_874
>> > []),SuperColumn(owneditem_884 []),SuperColumn(owneditem_886
>> > []),SuperColumn(owneditem_908 []),SuperColumn(owneditem_91
>> > []),SuperColumn(owneditem_911 []),SuperColumn(owneditem_930
>> > []),SuperColumn(owneditem_934 []),SuperColumn(owneditem_937
>> > []),SuperColumn(owneditem_944 []),SuperColumn(owneditem_945
>> > []),SuperColumn(owneditem_962 []),SuperColumn(owneditem_963
>> > []),SuperColumn(owneditem_964 []),])])
>> >         at
>> > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682)
>> >         at
>> >
>> > org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:279)
>> >         at
>> >
>> > org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:122)
>> >         at
>> >
>> > org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:599)
>> >         at
>> >
>> > org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
>> >         at
>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> >         at java.lang.Thread.run(Thread.java:662)
>> >  WARN 10:51:54,302 Heap is 0.764063958911146 full.  You may need to
>> > reduce
>> > memtable and/or cache sizes.  Cassandra will now flush up to the two
>> > largest
>> > memtables to free up memory.  Adjust flush_largest_memtables_at
>> > threshold in
>> > cassandra.yaml if you don't want Cassandra to do this automatically
>> >  WARN 10:51:54,303 Flushing CFS(Keyspace='Player',
>> > ColumnFamily='PlayerDetail') to relieve memory pressure
>> >  INFO 11:00:41,162 Started hinted handoff for token:
>> > 121529416757478022665490931225631504090 with IP: /10.16.96.212
>> >  INFO 11:00:41,163 Finished hinted handoff of 0 rows to endpoint
>> > /10.16.96.212
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor192]
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor165]
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor202]
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor232]
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor146]
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor181]
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor190]
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor214]
>> > ...
>> > [Unloading class sun.reflect.GeneratedConstructorAccessor19]
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor209]
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor227]
>> > [Unloading class
>> > sun.reflect.GeneratedSerializationConstructorAccessor205]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor202]
>> >
>> > The "Unloading" messages continued, with occasional mentions of hinted
>> > handoffs in between, until I killed Cassandra.
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message