I have SCF with ~250K rows. One of these rows is relatively
large - it's a wide row (according to compaction logs)
containing ~100.000 super columns and overall size of 1GB. Each
super column has average size of 10K and ~10 sub columns.
When I'm trying to delete ~90% of the columns in this particular
row, Cassandra nodes which own this wide row (3 of 5, RF=3)
quickly run out of the heap space. See logs from one of the
After that, all 3 nodes start flapping up/down and GC messages
(like the one in the bottom of the pastebin above) appearing in
the logs. Cassandra never repairs from this mode and the only
way out if to "kill -9" and start again. On IRC it was suggested
that it enters GC death spiral.
I tried to throttle delete requests on the client side - sending
batch of 100 delete requests each 500ms. So no more than 200
deletes/sec. But it didn't help. I can reduce it further to
100/sec, but I don't think it will help much.
I delete millions of columns from other row in this SCF at the
same rate and never have hit this problem. It only happens when
I try to delete from this particular wide row.
So right now I don't know how can I delete these columns. Any