Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: softfail (nike.apache.org: transitioning domain of
 andras.szerdahelyi@ignitionone.com does not designate 216.82.243.205 as
 permitted sender)
From: Andras Szerdahelyi <andras.szerdahelyi@ignitionone.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: 33million hinted handoffs from nowhere
Thread-Topic: 33million hinted handoffs from nowhere
Thread-Index: AQHOILdyl2cNzuTE1EeKr+jnH9CvSw==
Date: Thu, 14 Mar 2013 13:25:51 +0000
Message-ID: <FE84AE7AAE9A2B4EA73512EED142BEF830CAEC5B@ATL02MB02.corp.local>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.3.1.130117
Content-Type: multipart/alternative;
	boundary="_000_FE84AE7AAE9A2B4EA73512EED142BEF830CAEC5BATL02MB02corplo_"
MIME-Version: 1.0

--_000_FE84AE7AAE9A2B4EA73512EED142BEF830CAEC5BATL02MB02corplo_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

Hi list,

I am experiencing seemingly uncontrollable and unexplained growth of my Hin=
tedHandoff CF on a single node. Unexplained because there are no hinted han=
doffs being logged on the node, uncontrollable because I see 33 million ins=
erts in cfstats and the size of the stables is over 10 gigs all in an hour =
of uptime.


I have done the following to try and reproduce this:

- shut down my cluster
- on all nodes: remove sstables from the HintsColumnFamily data dir
- on all nodes: remove commit logs
- start all nodes but the one that=92s showing this problem
- nothing is writing to any of the nodes. There are no hinted handoff going=
 on anywhere
- bring back the node in question last
- few seconds after boot:

                Column Family: HintsColumnFamily
                SSTable count: 1
                Space used (live): 44946532
                Space used (total): 44946532
                Number of Keys (estimate): 256
                Memtable Columns Count: 17840
                Memtable Data Size: 17569909
                Memtable Switch Count: 2
                Read Count: 0
                Read Latency: NaN ms.
                Write Count: 184836
                Write Latency: 0.668 ms.
                Pending Tasks: 0
                Bloom Filter False Postives: 0
                Bloom Filter False Ratio: 0.00000
                Bloom Filter Space Used: 16
                Compacted row minimum size: 20924301
                Compacted row maximum size: 25109160
                Compacted row mean size: 25109160


--_000_FE84AE7AAE9A2B4EA73512EED142BEF830CAEC5BATL02MB02corplo_
Content-Type: text/html; charset="Windows-1252"
Content-ID: <536FA37F831AC446B2E613A4866CCD80@corp.local>
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1=
252">
</head>
<body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin=
e-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-fami=
ly: Calibri, sans-serif; ">
<div>Hi list,</div>
<div><br>
</div>
<div>I am experiencing seemingly uncontrollable and unexplained growth of m=
y HintedHandoff CF on a single node. Unexplained because there are no hinte=
d handoffs being logged on the node, uncontrollable because I see 33 millio=
n inserts in cfstats and the size
 of the stables is over 10 gigs all in an hour of uptime.&nbsp;</div>
<div><br>
</div>
<div><br>
</div>
<div>I have done the following to try and reproduce this:</div>
<div><br>
</div>
<div>- shut down my cluster</div>
<div>- on all nodes: remove sstables from the&nbsp;HintsColumnFamily&nbsp;d=
ata dir</div>
<div>- on all nodes: remove commit logs</div>
<div>- start all nodes but the one that=92s showing this problem</div>
<div>- nothing is writing to any of the nodes. There are no hinted handoff =
going on anywhere</div>
<div>- bring back the node in question last</div>
<div>- few seconds after boot:</div>
<div><br>
</div>
<div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Column Family:=
 HintsColumnFamily</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; SSTable count:=
 1</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Space used (li=
ve): 44946532</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Space used (to=
tal): 44946532</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Number of Keys=
 (estimate): 256</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Memtable Colum=
ns Count: 17840</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Memtable Data =
Size: 17569909</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Memtable Switc=
h Count: 2</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Read Count: 0<=
/div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Read Latency: =
NaN ms.</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Write Count: 1=
84836</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Write Latency:=
 0.668 ms.</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Pending Tasks:=
 0</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Bloom Filter F=
alse Postives: 0</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Bloom Filter F=
alse Ratio: 0.00000</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Bloom Filter S=
pace Used: 16</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Compacted row =
minimum size: 20924301</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Compacted row =
maximum size: 25109160</div>
<div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Compacted row =
mean size: 25109160</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</body>
</html>

--_000_FE84AE7AAE9A2B4EA73512EED142BEF830CAEC5BATL02MB02corplo_--