Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\))
Subject: Re: 1.1.11: system keyspace is filling up
From: Aaron Morton <aaron@thelastpickle.com>
In-Reply-To: 
 <CAJMN9iMStNmnLJrDaD7FA-BFxjFzdOCpqdFebLk59XZnoKWb1w@mail.gmail.com>
Date: Wed, 27 Nov 2013 14:47:41 +1300
Content-Transfer-Encoding: quoted-printable
Message-Id: <ACA3EB77-F70C-4872-9E30-8740AF91FB22@thelastpickle.com>
References: <l58ssf$b3u$1@ger.gmane.org>
 <CAEDUwd3m57Aj4aoZYGonnCNOH0XnsVOkrP=jhMe5NfnzTGTMmA@mail.gmail.com>
 <l5aq2c$gvv$1@ger.gmane.org>
 <CAJMN9iMStNmnLJrDaD7FA-BFxjFzdOCpqdFebLk59XZnoKWb1w@mail.gmail.com>
To: Cassandra User <user@cassandra.apache.org>

> What happens if they are not being successfully delivered ? Will they =
eventually TTL-out ?
They have a TTL set to the gc_grace_seconds on the CF at the time of the =
write.=20

I=92ve also seen hints build up in multi DC systems due to timeouts on =
the coordinator.  i.e. the remote nodes are up, co-ordinator starts the =
writes, remote nodes process the request (no dropped messages), but the =
response is lost. These are tracked as timeouts on the =
MessagingServiceMBean.=20

Cheers

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 22/11/2013, at 6:00 pm, Rahul Menon <rahul@apigee.com> wrote:

> Oleg,=20
>=20
> The system keyspace is not replicated it is local to the node. You =
should check your logs to see if there are Timeouts from streaming =
hints, i believe the default value to stream hints it 10 seconds. When i =
ran into this problem i truncated hints to clear out the space and then =
ran a repair so ensure that all the data was consistant across all =
nodes, even if there was a failure.=20
>=20
> -rm=20
>=20
>=20
> On Tue, Nov 5, 2013 at 6:29 PM, Oleg Dulin <oleg.dulin@gmail.com> =
wrote:
> What happens if they are not being successfully delivered ? Will they =
eventually TTL-out ?
>=20
>=20
>=20
> Also, do I need to truncate hints on every node or is it replicated ?
>=20
>=20
>=20
> Oleg
>=20
>=20
>=20
> On 2013-11-04 21:34:55 +0000, Robert Coli said:
>=20
>=20
>=20
> On Mon, Nov 4, 2013 at 11:34 AM, Oleg Dulin <oleg.dulin@gmail.com> =
wrote:
>=20
> I have a dual DC setup, 4 nodes, RF=3D4 in each.
>=20
>=20
>=20
> The one that is used as primary has its system keyspace fill up with =
200 gigs of data, majority of which is hints.
>=20
>=20
>=20
> Why does this happen ?
>=20
>=20
>=20
> How can I clean it up ?
>=20
>=20
>=20
> If you have this many hints, you probably have flapping / frequent =
network partition, or very overloaded nodes. If you compare the number =
of hints to the number of dropped messages, that would be informative. =
If you're hinting because you're dropping, increase capacity. If you're =
hinting because of partition, figure out why there's so much partition.
>=20
>=20
>=20
> WRT cleaning up hints, they will automatically be cleaned up =
eventually, as long as they are successfully being delivered. If you =
need to manually clean them up you can truncate system.hints keyspace.
>=20
>=20
>=20
> =3DRob
>=20
> =20
>=20
>=20
>=20
>=20
>=20
>=20
> --=20
>=20
> Regards,
>=20
> Oleg Dulin
>=20
> http://www.olegdulin.com
>=20
>=20