Sure I can fill in the ticket. Here is what I have noticed so far, the count of HH is not going up, which is good. I think what must have happened is that after I restarted the cluster, no new hints were added just the old one's are still around and not cleaned up, is that possible? Cannot say for sure, since I only looked at this JMX bean abt 36 hours after restart.
Can I just clean this up using the JMX call? I do not want to turn off HH, since that can handle the intermittent network hiccups well, right?
Could you put together some information on this in a ticket and references this one https://issues.apache.org/jira/browse/CASSANDRA-3071The short term fix is to disable HH. You will still get consistent reads.CheersOn 25/08/2011, at 3:22 AM, Anand Somani wrote:So I have looked at the cluster from
- Cassandra-client - describe cluster => shows correctly - 3 nodes
- used the StorageService - JMX bean =>UnreachableNodes - shows 0
If all these show the correct ring state, why are hints being maintained, looks like that is the only way to find out about "phantom" nodes.
On Wed, Aug 24, 2011 at 8:01 AM, Anand Somani <firstname.lastname@example.org> wrote:So, I restarted the cluster (not rolling), but it is still maintaining hints for the IP's that are no longer part of the ring. nodetool ring shows things correctly (as only 3 nodes).
When I check thru the jmx hintedhandoff manager, it shows it is maintaining the hints for those non existent IP's. So the question is
- How can I remove these IP permanently, so hints do not get saved?
- Not all nodes see the same list of IP'sOn Sun, Aug 21, 2011 at 3:10 PM, aaron morton <email@example.com> wrote:
Yup, you can check the what HH is doing via JMX.
there is a bug in 0.7 that can result in log files not been deleted https://issues.apache.org/jira/browse/CASSANDRA-2829CheersOn 22/08/2011, at 4:56 AM, Anand Somani wrote:We have a lot of space on /data, and looks like it was flushing data fine from file timestamps.
We did have a bit of goofup with IP's when bringing up a down node (and the commit files have been around since then). Wonder if that is what triggered it and we have a bunch of hinted handoff's being backed up.
For hinted hand off - how do I check if the nodes are collecting hints ( I do have it turned on). I noticed console bean HintedHandManager, is that the only way to find out.On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller <firstname.lastname@example.org> wrote:
> When does the actual commit-data file get deleted.They *should* be getting deleted when they no longer contain any data
> The flush interval on all my memtables is 60 minutes
that has not been flushed to disk. Are flushes definitely still
happening? Is it possible flushing has started failing (e.g. out of
The only way I can think of over nodes directly affecting the commit
log size on your node would be e.g. hinted handoff resulting in burst
/ Peter Schuller (@scode on twitter)