incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andras Szerdahelyi <andras.szerdahe...@ignitionone.com>
Subject Re: 33million hinted handoffs from nowhere
Date Thu, 14 Mar 2013 13:30:43 GMT
( The previous letter was sent prematurely, sorry. )

This node is the only node being written to, but the Cfs being written replicate to almost
all of the other nodes
My understanding is that hinted handoff is mutations kept around on the coordinator node,
to be replayed when the target node re-appears on the ring. All my nodes are up and again,
no hinted handoff is logged on the node itself

Thanks!
Andras

From: Andras Szerdahelyi <andras.szerdahelyi@ignitionone.com<mailto:andras.szerdahelyi@ignitionone.com>>
Date: Thursday 14 March 2013 14:25
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: 33million hinted handoffs from nowhere

Hi list,

I am experiencing seemingly uncontrollable and unexplained growth of my HintedHandoff CF on
a single node. Unexplained because there are no hinted handoffs being logged on the node,
uncontrollable because I see 33 million inserts in cfstats and the size of the stables is
over 10 gigs all in an hour of uptime.


I have done the following to try and reproduce this:

- shut down my cluster
- on all nodes: remove sstables from the HintsColumnFamily data dir
- on all nodes: remove commit logs
- start all nodes but the one that’s showing this problem
- nothing is writing to any of the nodes. There are no hinted handoff going on anywhere
- bring back the node in question last
- few seconds after boot:

                Column Family: HintsColumnFamily
                SSTable count: 1
                Space used (live): 44946532
                Space used (total): 44946532
                Number of Keys (estimate): 256
                Memtable Columns Count: 17840
                Memtable Data Size: 17569909
                Memtable Switch Count: 2
                Read Count: 0
                Read Latency: NaN ms.
                Write Count: 184836
                Write Latency: 0.668 ms.
                Pending Tasks: 0
                Bloom Filter False Postives: 0
                Bloom Filter False Ratio: 0.00000
                Bloom Filter Space Used: 16
                Compacted row minimum size: 20924301
                Compacted row maximum size: 25109160
                Compacted row mean size: 25109160





Mime
View raw message