What happens if they are not being successfully delivered ? Will they eventually TTL-out ?
Also, do I need to truncate hints on every node or is it replicated ?
On 2013-11-04 21:34:55 +0000, Robert Coli said:
On Mon, Nov 4, 2013 at 11:34 AM, Oleg Dulin <email@example.com> wrote:
I have a dual DC setup, 4 nodes, RF=4 in each.
The one that is used as primary has its system keyspace fill up with 200 gigs of data, majority of which is hints.
Why does this happen ?
How can I clean it up ?
If you have this many hints, you probably have flapping / frequent network partition, or very overloaded nodes. If you compare the number of hints to the number of dropped messages, that would be informative. If you're hinting because you're dropping, increase capacity. If you're hinting because of partition, figure out why there's so much partition.
WRT cleaning up hints, they will automatically be cleaned up eventually, as long as they are successfully being delivered. If you need to manually clean them up you can truncate system.hints keyspace.