cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra and massive TTL expirations cause HEAP issue
Date Mon, 02 Jul 2012 23:33:34 GMT
>  After 10 days my cluster crashes due to a java.lang.OutOfMemoryError during compaction
of the big column family that contains roughly 95% of the data. 

Does this column family have very wide rows ? 

>  simply some tweaks I need to make in the yaml file.  I have tried:
The main things that reduce the impact compaction has on memory are:

in_memory_compaction_limit_in_mb
concurrent_compactors

Of the top of my head I cannot think of any shortcuts taken by compaction if/when all data
in an SSTable is past TTL. I think there was talk of something like that though. 

Hope that helps.

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 27/06/2012, at 2:38 AM, Nils Pommerien wrote:

> Hello,
> I am evaluating Cassandra in a log retrieval application.  My ring conists of3 m2.xlarge
instances (17.1 GB memory, 6.5 ECU (2 virtual cores with 3.25 EC2 Compute Units each), 420
GB of local instance storage, 64-bit platform) and I am writing at roughly 220 writes/sec.
 Per day I am adding roughly 60GB of data.  All of this sounds simple and easy and all three
nodes are humming along with basically no load.  
> 
> The issue is that I am writing all my data with a TTL of 10 days.  After 10 days my cluster
crashes due to a java.lang.OutOfMemoryError during compaction of the big column family that
contains roughly 95% of the data.  So basically after 10 days my data set is 600GB and after
10 days Cassandra would have to tombstone and purge 60GB of data at the same rate of roughly
220 deletes/second.  I am not sure if Cassandra should be able to do it, whether I should
take a partitioning approach (one CF per day), or if there is simply some tweaks I need to
make in the yaml file.  I have tried:
> Decrease flush-largest-memtables-at to .4 
> reduce_cache_sizes_at and reduce_cache_capacity_to set to 1
> Now, the issue remains the same:
> 
> WARN [ScheduledTasks:1] 2012-06-11 19:39:42,017 GCInspector.java (line 145) Heap is 0.9920103380107628
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically.
> 
> Eventually it will just die with this message.  This affects all nodes in the cluster,
not just one. 
>  
> Dump file is incomplete: file size limit
> ERROR 19:39:39,695 Exception in thread Thread[ReadStage:134,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR 19:39:39,724 Exception in thread Thread[MutationStage:57,5,main]
> java.lang.OutOfMemoryError: Java heap space
>       at org.apache.cassandra.utils.FBUtilities.hashToBigInteger(FBUtilities.java:213)
>       at org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:154)
>       at org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.java:47)
>       at org.apache.cassandra.db.RowPosition.forKey(RowPosition.java:54)
>  
> Any help is highly appreciated.  It would be cool to tweak it in a way that I can have
a moving window of 10 days in Cassandra while dropping the old data… Or, if there is any
other recommended way to deal with such sliding time windows I am open for ideas.
> 
> Thank you for your help!		


Mime
View raw message