cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rick Gunderson" <>
Subject tombstone_failure_threshold being ignored?
Date Thu, 28 Apr 2016 17:22:59 GMT
We are running Cassandra 2.2.3, 2 data centers, 3 nodes in each. The 
replication factor per datacenter is 3. The Xmx setting on the Cassandra 
JVMs is 4GB.

We have a workload that generates loots of tombstones and Cassandra goes 
OOM in about 24 hours. We've adjusted the tombstone_failure_threshold down 
to 25000 but we never see the TombstoneOverwhelmingException before the 
nodes start going OOM.

The table operation that looks to be the culprit is a scan of partition 
keys (i.e. we are scanning across narrow rows, not scanning within a wide 
row). The heapdump shows we have a RangeSliceReply containing an ArrayList 
with 1,823,230 org.apache.cassandra.db.Row objects with a retained heap 
size of 441MiB.  A look inside one of the Row objects shows an 
org.apache.cassandra.db.DeletionInfo object so I assume that means the row 
has been tombstoned.

If all of the 1,823,239 Row objects are tombstoned (and it is likely that 
most of them are), is there a reason that the 
TombstoneOverwhelmingException never gets thrown? 


Rick (R.) Gunderson 
Software Engineer
IBM Commerce, B2B Development - GDHA

Phone: 1-250-220-1053 
Find me on:  

1803 Douglas St
Victoria, BC V8T 5C3 

View raw message