incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Michalski <mich...@opera.com>
Subject Re: Unable to drop secondary index
Date Wed, 24 Apr 2013 12:27:39 GMT
Not really sure if it has something to do with the schema problems, but 
I think the fact that node was down caused us to hit 
https://issues.apache.org/jira/browse/CASSANDRA-5179 (a bit different 
output on sender's side, but looks similar in general) - after checking 
logs with debug level TRACE it seems that all these "add mutation & 
compact forever" is caused by "infinite" HH from some of the nodes. Even 
if it's not a root cause, it still a little step forward ;-)

BTW. Looks eeevil!

cqlsh:system> SELECT * FROM hints limit 1;
Request did not complete within rpc_timeout.

root@cssa02-13:~# ls -lahS /cassandra/system/hints/ | grep "Data\." | wc -l
83

I don't even want to think what's inside ;-)

M.

W dniu 24.04.2013 08:54, Michal Michalski pisze:
>
>> The log messages seem fine to me. It's handling eventually updating
>> the secondary indexes.
>
> Yes, but now it also applies mutations, while (I'm 100% sure) nothing
> was written to the affected CFs ;-)
>
> DEBUG [MutationStage:87] 2013-04-24 06:41:45,695
> RowMutationVerbHandler.java (line 40) Applying mutation
> DEBUG [MutationStage:56] 2013-04-24 06:41:45,695
> RowMutationVerbHandler.java (line 40) Applying mutation
> DEBUG [MutationStage:123] 2013-04-24 06:41:45,695
> RowMutationVerbHandler.java (line 40) Applying mutation
> DEBUG [MutationStage:118] 2013-04-24 06:41:45,695
>
> And it's happening for > 24 hours now with about 1000 mutations / sec
> (with no writes from clients!), for only 3 CFs (and as I said, this CFs
> are between 2 and 80 MB large), causing non-stop SSTable
> creation/compaction - see attached screen.
>
> Strange, but at least the node is more or less "fine" in general (and
> the second one is 100% fine) ;-)
>
> Anyway, my original schema problem is still there and I can't update
> anything, but I'm not going to make an attempt to fix it once again 2
> days before my holidays, so I think I'll get back to this problem in the
> end of May ;-)
>
> Thanks for help :-)
>
> M.
>


Mime
View raw message