cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andras Szerdahelyi <>
Subject 33million hinted handoffs from nowhere
Date Thu, 14 Mar 2013 13:25:51 GMT
Hi list,

I am experiencing seemingly uncontrollable and unexplained growth of my HintedHandoff CF on
a single node. Unexplained because there are no hinted handoffs being logged on the node,
uncontrollable because I see 33 million inserts in cfstats and the size of the stables is
over 10 gigs all in an hour of uptime.

I have done the following to try and reproduce this:

- shut down my cluster
- on all nodes: remove sstables from the HintsColumnFamily data dir
- on all nodes: remove commit logs
- start all nodes but the one that’s showing this problem
- nothing is writing to any of the nodes. There are no hinted handoff going on anywhere
- bring back the node in question last
- few seconds after boot:

                Column Family: HintsColumnFamily
                SSTable count: 1
                Space used (live): 44946532
                Space used (total): 44946532
                Number of Keys (estimate): 256
                Memtable Columns Count: 17840
                Memtable Data Size: 17569909
                Memtable Switch Count: 2
                Read Count: 0
                Read Latency: NaN ms.
                Write Count: 184836
                Write Latency: 0.668 ms.
                Pending Tasks: 0
                Bloom Filter False Postives: 0
                Bloom Filter False Ratio: 0.00000
                Bloom Filter Space Used: 16
                Compacted row minimum size: 20924301
                Compacted row maximum size: 25109160
                Compacted row mean size: 25109160

View raw message