incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rustam Aliyev <>
Subject Deleting from SCF wide row makes node unresponsive
Date Sat, 02 Jun 2012 07:43:48 GMT
 Hi all,

I have SCF with ~250K rows. One of these rows is relatively large - it's a
wide row (according to compaction logs) containing ~100.000 super columns
and overall size of 1GB. Each super column has average size of 10K and ~10
sub columns.

When I'm trying to delete ~90% of the columns in this particular row,
Cassandra nodes which own this wide row (3 of 5, RF=3) quickly run out of
the heap space. See logs from one of the hosts here:

After that, all 3 nodes start flapping up/down and GC messages (like the
one in the bottom of the pastebin above) appearing in the logs. Cassandra
never repairs from this mode and the only way out if to "kill -9" and start
again. On IRC it was suggested that it enters GC death spiral.

I tried to throttle delete requests on the client side - sending batch of
100 delete requests each 500ms. So no more than 200 deletes/sec. But it
didn't help. I can reduce it further to 100/sec, but I don't think it will
help much.

I delete millions of columns from other row in this SCF at the same rate
and never have hit this problem. It only happens when I try to delete from
this particular wide row.

So right now I don't know how can I delete these columns. Any ideas?

Many thanks,

View raw message