See https://issues.apache.org/jira/browse/CASSANDRA-3741
I did post a fix there that helped me.

2012/4/24 crypto five <cryptofive@gmail.com>
Hi,

I have 50 millions of rows in column family on 4G RAM box. I allocatedf 2GB to cassandra.
I have program which is traversing this CF and cleaning some data there, it generates about 20k delete statements per second.
After about of 3 millions deletions cassandra stops responding to queries: it doesn't react to CLI, nodetool etc.
I see in the logs that it tries to free some memory but can't even if I wait whole day. 
Also I see following in  the logs:

INFO [ScheduledTasks:1] 2012-04-23 18:38:13,333 StorageService.java (line 2647) Unable to reduce heap usage since there are no dirty column families

When I am looking at memory dump I see that memory goes to ConcurrentSkipListMap(10%), HeapByteBuffer(13%), DecoratedKey(6%), int[](6%), BigInteger(8.2%), ConcurrentSkipListMap$HeadIndex(7.2%), ColumnFamily(6.5%), ThreadSafeSortedColumns(13.7%), long[](5.9%).

What can I do to make cassandra stop dying? 
Why it can't free the memory?
Any ideas?

Thank you.



--
Best regards,
 Vitalii Tymchyshyn