incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul Menon <ra...@apigee.com>
Subject Re: 1.1.11: system keyspace is filling up
Date Fri, 22 Nov 2013 05:00:35 GMT
Oleg,

The system keyspace is not replicated it is local to the node. You should
check your logs to see if there are Timeouts from streaming hints, i
believe the default value to stream hints it 10 seconds. When i ran into
this problem i truncated hints to clear out the space and then ran a repair
so ensure that all the data was consistant across all nodes, even if there
was a failure.

-rm


On Tue, Nov 5, 2013 at 6:29 PM, Oleg Dulin <oleg.dulin@gmail.com> wrote:

>  What happens if they are not being successfully delivered ? Will they
> eventually TTL-out ?
>
>
> Also, do I need to truncate hints on every node or is it replicated ?
>
>
> Oleg
>
>
> On 2013-11-04 21:34:55 +0000, Robert Coli said:
>
>
> On Mon, Nov 4, 2013 at 11:34 AM, Oleg Dulin <oleg.dulin@gmail.com> wrote:
>
> I have a dual DC setup, 4 nodes, RF=4 in each.
>
>
> The one that is used as primary has its system keyspace fill up with 200
> gigs of data, majority of which is hints.
>
>
> Why does this happen ?
>
>
> How can I clean it up ?
>
>
> If you have this many hints, you probably have flapping / frequent network
> partition, or very overloaded nodes. If you compare the number of hints to
> the number of dropped messages, that would be informative. If you're
> hinting because you're dropping, increase capacity. If you're hinting
> because of partition, figure out why there's so much partition.
>
>
> WRT cleaning up hints, they will automatically be cleaned up eventually,
> as long as they are successfully being delivered. If you need to manually
> clean them up you can truncate system.hints keyspace.
>
>
> =Rob
>
>
>
>
>
>
> --
>
> Regards,
>
> Oleg Dulin
>
> http://www.olegdulin.com
>

Mime
View raw message