cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolai Gylling>
Subject Sudden increase in diskspace usage
Date Thu, 09 May 2013 05:43:00 GMT

I have a 3-node SSD-based cluster, with around 1 TB data, RF:3, C* v.1.2.0,
vnodes. One large CF, LCS. Everything was running smooth, until one of the
nodes crashed and was restarted.

At the time of normal operation there was 800 gb free space on each node.
After the crash, C* started using a lot more, resulting in an
out-of-diskspace situation on 2 nodes, eg. C* used up the 800 gb in just 2
days, giving us very little time to do anything about it, since
repairs/joins takes a considerable amount of time.

What can make C* suddenly use this amount of disk-space? We did see a lot
of pending compactions on one node (7k).

Any tips on recovering from an out-of-diskspace on multiple nodes,
situation? I've tried moving some SStables away, but C* seems to use
whatever space I free up in no time. I'm not sure if any of the nodes is
fully updated as 'nodetool status' reports 3 different loads

--  Address           Load       Tokens  Owns (effective)  Host ID
UN     1.4 TB     256     100.0%
 1261717d-ddc1-457e-9c93-431b3d3b5c5b  rack1
UN    1.03 TB    256     100.0%
 f80bfa31-e19d-4346-9a14-86ae87f06356  rack1
DN      1.11 TB    256     100.0%
 85d4cd28-93f4-4b96-8140-3605302e90a9  rack1



*Nicolai Gylling*

View raw message