So, I restarted the cluster (not rolling), but it is still maintaining hints for the IP's that are no longer part of the ring. nodetool ring shows things correctly (as only 3 nodes).
When I check thru the jmx hintedhandoff manager, it shows it is maintaining the hints for those non existent IP's. So the question is
 - How can I remove these IP permanently, so hints do not get saved?
 - Not all nodes see the same list of IP's



On Sun, Aug 21, 2011 at 3:10 PM, aaron morton <aaron@thelastpickle.com> wrote:
Yup, you can check the what HH is doing via JMX.

there is a bug in 0.7 that can result in log files not been deleted https://issues.apache.org/jira/browse/CASSANDRA-2829

Cheers
 
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton

On 22/08/2011, at 4:56 AM, Anand Somani wrote:

We have a lot of space on /data, and looks like it was flushing data fine from file timestamps.

We did have a bit of goofup with IP's when bringing up a down node (and the commit files have been around since then). Wonder if that is what triggered it and we have a bunch of hinted handoff's being backed up.

For hinted hand off - how do I check if the nodes are collecting hints ( I do have it turned on). I noticed console bean HintedHandManager, is that the only way to find out.

On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller <peter.schuller@infidyne.com> wrote:
> When does the actual commit-data file get deleted.
>
> The flush interval on all my memtables is 60 minutes

They *should* be getting deleted when they no longer contain any data
that has not been flushed to disk. Are flushes definitely still
happening? Is it possible flushing has started failing (e.g. out of
disk)?

The only way I can think of over nodes directly affecting the commit
log size on your node would be e.g. hinted handoff resulting in burst
of writes.

--
/ Peter Schuller (@scode on twitter)