Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted
 nor denied by domain of mlarsson@yahoo-inc.com)
From: Mattias Larsson <mlarsson@yahoo-inc.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Fri, 26 Oct 2012 11:56:08 -0700
Subject: Re: Hinted Handoff storage inflation
Thread-Topic: Hinted Handoff storage inflation
Thread-Index: Ac2zq4+RmwE5+B2ZQHWaZvoGcOXreg==
Message-ID: <ED100894-3A43-4116-A8BB-1038E5974CF3@yahoo-inc.com>
References: <1C115F46-280B-495B-B361-1D6EFE059842@yahoo-inc.com>
 <DC2CD3BE-5503-4116-81C1-E12F3D08D795@thelastpickle.com>
In-Reply-To: <DC2CD3BE-5503-4116-81C1-E12F3D08D795@thelastpickle.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Oct 24, 2012, at 6:05 PM, aaron morton wrote:

> Hints store the columns, row key, KS name and CF id(s) for each mutation =
to each node. Where an executed mutation will store the most recent columns=
 collated with others under the same row key. So depending on the type of m=
utation hints will take up more space.=20
>=20
> The worse case would be lots of overwrites. After that writing a small am=
ount of data to many rows would result in a lot of the serialised space bei=
ng devoted to row keys, KS name and CF id.
>=20
> 16Gb is a lot though. What was the write workload like ?

Each write is new data only (no overwrites). Each mutation adds a row to on=
e column family with a column containing about ~100 bytes of data and a new=
 row to another column family with a SuperColumn containing 2x17KiB payload=
s. These are sent in batches with several in them, but I found that the sto=
rage overhead was the same regardless of the size of the batch mutation (i.=
e., 5 vs 25 mutations made no difference). A total of 1,000,000 mutations l=
ike these are sent over the duration of the test.


> You can get an estimate on the number of keys in the Hints CF using nodet=
ool cfstats. Also some metrics in the JMX will tell you how many hints are =
stored.=20
>=20
>> This has a huge impact on write performance as well.
> Yup. Hints are added to the same Mutation thread pool as normal mutations=
. They are processed async to the mutation request but they still take reso=
urces to store.=20
>=20
> You can adjust how long hints a collected for with max_hint_window_in_ms =
in the yaml file.=20
>=20
> How long did the test run for ?=20
>=20

With both data centers functional, the test takes just a few minutes to run=
, with one data center down, 15x the amount of time.

/dml