You can drop the hints via JMX and stopping the node and deleting the SSTables. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton

On 25/04/2013, at 12:27 AM, Michal Michalski <michalm@opera.com> wrote:

Not really sure if it has something to do with the schema problems, but I think the fact that node was down caused us to hit https://issues.apache.org/jira/browse/CASSANDRA-5179 (a bit different output on sender's side, but looks similar in general) - after checking logs with debug level TRACE it seems that all these "add mutation & compact forever" is caused by "infinite" HH from some of the nodes. Even if it's not a root cause, it still a little step forward ;-)

BTW. Looks eeevil!

cqlsh:system> SELECT * FROM hints limit 1;
Request did not complete within rpc_timeout.

root@cssa02-13:~# ls -lahS /cassandra/system/hints/ | grep "Data\." | wc -l
83

I don't even want to think what's inside ;-)

M.

W dniu 24.04.2013 08:54, Michal Michalski pisze:

The log messages seem fine to me. It's handling eventually updating
the secondary indexes.

Yes, but now it also applies mutations, while (I'm 100% sure) nothing
was written to the affected CFs ;-)

DEBUG [MutationStage:87] 2013-04-24 06:41:45,695
RowMutationVerbHandler.java (line 40) Applying mutation
DEBUG [MutationStage:56] 2013-04-24 06:41:45,695
RowMutationVerbHandler.java (line 40) Applying mutation
DEBUG [MutationStage:123] 2013-04-24 06:41:45,695
RowMutationVerbHandler.java (line 40) Applying mutation
DEBUG [MutationStage:118] 2013-04-24 06:41:45,695

And it's happening for > 24 hours now with about 1000 mutations / sec
(with no writes from clients!), for only 3 CFs (and as I said, this CFs
are between 2 and 80 MB large), causing non-stop SSTable
creation/compaction - see attached screen.

Strange, but at least the node is more or less "fine" in general (and
the second one is 100% fine) ;-)

Anyway, my original schema problem is still there and I can't update
anything, but I'm not going to make an attempt to fix it once again 2
days before my holidays, so I think I'll get back to this problem in the
end of May ;-)

Thanks for help :-)

M.