cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Hatfield <nick.hatfi...@metricly.com>
Subject Re: can i...
Date Thu, 07 Mar 2019 17:08:25 GMT
Amazing thank you!

I have an overlapping timestamp data issue in multiple SSTables. This is a production cluster
of 27 nodes using Kairosdb front-end on cassandra 3.11. Our kairosdb data_points table, looks
like this:

CREATE TABLE kairosdb.data_points (
    key blob,
    column1 blob,
    value blob,
    PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE
    AND CLUSTERING ORDER BY (column1 ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'com.jeffjirsa.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '1', 'compaction_window_unit': 'DAYS', 'max_threshold': '32', 'min_threshold':
'4', 'timestamp_resolution': 'MILLISECONDS', 'tombstone_compaction_interval': '432000', 'tombstone_threshold':
'0.2', 'unchecked_tombstone_compaction': 'true'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 7884009
    AND gc_grace_seconds = 432000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.1
    AND speculative_retry = '99PERCENTILE’;


I run repairs once every 3 days on this 27 node cluster. I ran a manual compaction on individual
sstables that had very old timestamps. It seems that all that did was merge the old stale
data, into a less sstables, rather than deleting the tombstoned data completely. Here’s
a quick look at a few entries from the sstable metadata:


Max: 01/01/2019 Min: 12/31/2018 Estimated droppable tombstones: 0.6407875588103883       
6.8G Mar 4 22:40 mc-230833-big-Data.db
Max: 01/02/2019 Min: 01/01/2019 Estimated droppable tombstones: 0.6717275763312848       
8.7G Mar 5 12:38 mc-231449-big-Data.db
Max: 01/03/2019 Min: 01/02/2019 Estimated droppable tombstones: 0.7037095860184683       
14G Mar 5 14:01 mc-231502-big-Data.db
Max: 01/04/2019 Min: 01/03/2019 Estimated droppable tombstones: 0.7081188342023961       
17G Mar 5 02:03 mc-230946-big-Data.db
Max: 01/05/2019 Min: 01/04/2019 Estimated droppable tombstones: 0.7148984115492688       
21G Mar 5 05:06 mc-231068-big-Data.db
Max: 01/06/2019 Min: 01/05/2019 Estimated droppable tombstones: 0.7190351559091802       
21G Mar 5 10:40 mc-231315-big-Data.db
Max: 01/07/2019 Min: 01/06/2019 Estimated droppable tombstones: 0.7174924163467192       
22G Mar 4 20:05 mc-230680-big-Data.db
Max: 01/08/2019 Min: 01/07/2019 Estimated droppable tombstones: 0.7209061510375004       
22G Mar 5 09:02 mc-231243-big-Data.db
Max: 01/09/2019 Min: 01/08/2019 Estimated droppable tombstones: 0.7152589769956947       
19G Mar 4 20:02 mc-230685-big-Data.db
Max: 01/10/2019 Min: 01/09/2019 Estimated droppable tombstones: 0.6791591497664088       
8.0G Mar 5 04:43 mc-231091-big-Data.db
Max: 01/11/2019 Min: 01/10/2019 Estimated droppable tombstones: 0.6903846423101958       
12G Mar 5 15:58 mc-231600-big-Data.db
Max: 01/12/2019 Min: 01/11/2019 Estimated droppable tombstones: 0.7960901678466651       
14G Mar 5 05:48 mc-231118-big-Data.db
Max: 01/13/2019 Min: 01/12/2019 Estimated droppable tombstones: 0.7925577980175544       
13G Mar 5 18:45 mc-231725-big-Data.db
Max: 01/14/2019 Min: 01/13/2019 Estimated droppable tombstones: 0.7977456322563183       
14G Mar 5 15:38 mc-231577-big-Data.db
Max: 01/15/2019 Min: 01/14/2019 Estimated droppable tombstones: 0.7914742161174189       
14G Mar 5 17:39 mc-231674-big-Data.db
Max: 01/16/2019 Min: 01/15/2019 Estimated droppable tombstones: 0.7844429396813951       
13G Mar 5 11:07 mc-231363-big-Data.db
Max: 01/17/2019 Min: 01/16/2019 Estimated droppable tombstones: 0.5902276606951279       
6.6G Mar 5 02:13 mc-230984-big-Data.db
Max: 01/18/2019 Min: 01/17/2019 Estimated droppable tombstones: 0.6547576878709388       
7.4G Mar 5 00:57 mc-230924-big-Data.db
Max: 01/19/2019 Min: 01/18/2019 Estimated droppable tombstones: 0.6892596936899507       
8.0G Mar 4 19:59 mc-230714-big-Data.db
Max: 01/20/2019 Min: 01/19/2019 Estimated droppable tombstones: 0.7076946624407203       
8.2G Mar 5 19:14 mc-231761-big-Data.db
Max: 01/21/2019 Min: 01/20/2019 Estimated droppable tombstones: 0.7112744492848729       
8.8G Mar 5 11:53 mc-231418-big-Data.db
Max: 01/22/2019 Min: 01/21/2019 Estimated droppable tombstones: 0.7098184906956251       
9.0G Mar 5 12:59 mc-231466-big-Data.db
Max: 01/23/2019 Min: 01/22/2019 Estimated droppable tombstones: 0.7280378416128286       
9.4G Mar 4 20:04 mc-230712-big-Data.db
Max: 01/24/2019 Min: 01/23/2019 Estimated droppable tombstones: 0.7375989189486377       
9.9G Mar 5 08:03 mc-231233-big-Data.db
Max: 01/25/2019 Min: 01/24/2019 Estimated droppable tombstones: 0.7550665105646288       
11G Mar 5 05:54 mc-231130-big-Data.db
Max: 01/26/2019 Min: 01/25/2019 Estimated droppable tombstones: 0.755651643899716        
11G Mar 5 17:19 mc-231662-big-Data.db
Max: 01/27/2019 Min: 01/26/2019 Estimated droppable tombstones: 0.7737130829029405       
12G Mar 5 08:23 mc-231245-big-Data.db
Max: 01/28/2019 Min: 01/27/2019 Estimated droppable tombstones: 0.7698542540694441       
12G Mar 5 19:47 mc-231774-big-Data.db
Max: 01/29/2019 Min: 01/28/2019 Estimated droppable tombstones: 0.7751910087975257       
13G Mar 5 04:25 mc-231064-big-Data.db
Max: 01/30/2019 Min: 01/29/2019 Estimated droppable tombstones: 0.7808428227545806       
13G Mar 5 18:38 mc-231719-big-Data.db
Max: 01/31/2019 Min: 01/30/2019 Estimated droppable tombstones: 0.7833226924138593       
14G Mar 5 02:26 mc-230969-big-Data.db
Max: 02/01/2019 Min: 01/31/2019 Estimated droppable tombstones: 0.7857791249217135       
14G Mar 5 19:10 mc-231733-big-Data.db
Max: 02/02/2019 Min: 02/01/2019 Estimated droppable tombstones: 0.7866910479817485       
17G Mar 5 06:54 mc-231166-big-Data.db
Max: 02/03/2019 Min: 02/02/2019 Estimated droppable tombstones: 0.5542345959736666       
18G Mar 5 11:45 mc-231396-big-Data.db
Max: 02/04/2019 Min: 02/03/2019 Estimated droppable tombstones: 0.5119360095151949       
19G Mar 5 03:28 mc-231022-big-Data.db
Max: 02/05/2019 Min: 02/04/2019 Estimated droppable tombstones: 0.2738841238834289       
19G Mar 5 10:11 mc-231322-big-Data.db
Max: 02/06/2019 Min: 02/05/2019 Estimated droppable tombstones: 0.2613869586284185       
19G Mar 5 11:35 mc-231388-big-Data.db
Max: 02/07/2019 Min: 02/06/2019 Estimated droppable tombstones: 0.16169525425765702      
19G Mar 6 18:39 mc-232699-big-Data.db
Max: 02/08/2019 Min: 02/07/2019 Estimated droppable tombstones: 0.1705846230960788       
18G Mar 6 19:59 mc-232756-big-Data.db
Max: 02/09/2019 Min: 02/08/2019 Estimated droppable tombstones: 0.20397309825287832      
18G Mar 5 06:14 mc-231141-big-Data.db
Max: 02/10/2019 Min: 02/09/2019 Estimated droppable tombstones: 0.21416796761523366      
18G Mar 5 05:46 mc-231119-big-Data.db
Max: 02/11/2019 Min: 02/10/2019 Estimated droppable tombstones: 0.2167303814761698       
19G Mar 5 13:20 mc-231468-big-Data.db
Max: 02/12/2019 Min: 02/11/2019 Estimated droppable tombstones: 0.2125492567433847       
19G Mar 5 14:25 mc-231520-big-Data.db
Max: 02/13/2019 Min: 02/12/2019 Estimated droppable tombstones: 0.21271803514892473      
19G Mar 5 15:17 mc-231565-big-Data.db
Max: 02/14/2019 Min: 02/13/2019 Estimated droppable tombstones: 0.223317411505499        
19G Mar 5 12:52 mc-231443-big-Data.db
Max: 02/15/2019 Min: 02/14/2019 Estimated droppable tombstones: 0.2380465637337778       
19G Mar 5 07:14 mc-231185-big-Data.db
Max: 02/16/2019 Min: 02/15/2019 Estimated droppable tombstones: 0.24416048715449007      
19G Mar 4 23:06 mc-230822-big-Data.db
Max: 02/17/2019 Min: 02/16/2019 Estimated droppable tombstones: 0.24840416482860103      
19G Mar 6 15:23 mc-232556-big-Data.db
Max: 02/18/2019 Min: 02/17/2019 Estimated droppable tombstones: 0.2388316490362356       
19G Mar 6 15:51 mc-232578-big-Data.db
Max: 02/19/2019 Min: 02/18/2019 Estimated droppable tombstones: 0.24455869859040452      
20G Mar 6 16:20 mc-232596-big-Data.db
Max: 02/20/2019 Min: 02/19/2019 Estimated droppable tombstones: 0.22053576261936442      
21G Mar 6 16:49 mc-232619-big-Data.db
Max: 02/21/2019 Min: 02/20/2019 Estimated droppable tombstones: 0.12749878111824647      
21G Mar 6 17:17 mc-232637-big-Data.db
Max: 02/22/2019 Min: 02/21/2019 Estimated droppable tombstones: 0.01626749369647037      
21G Mar 6 17:45 mc-232660-big-Data.db
Max: 02/23/2019 Min: 02/22/2019 Estimated droppable tombstones: 0.09138094337300502      
21G Mar 6 18:13 mc-232676-big-Data.db
Max: 02/24/2019 Min: 02/23/2019 Estimated droppable tombstones: 0.0       21G Mar 6 19:06
mc-232719-big-Data.db
Max: 02/25/2019 Min: 02/24/2019 Estimated droppable tombstones: 3.539084062959639E-8     
22G Mar 6 19:35 mc-232736-big-Data.db
Max: 02/26/2019 Min: 02/25/2019 Estimated droppable tombstones: 1.5737638608259986E-8    
22G Mar 6 20:28 mc-232772-big-Data.db
Max: 02/27/2019 Min: 02/26/2019 Estimated droppable tombstones: 1.2888956685492647E-8    
22G Mar 6 20:57 mc-232796-big-Data.db
Max: 02/28/2019 Min: 02/27/2019 Estimated droppable tombstones: 6.44921720903649E-9      
22G Mar 1 01:12 mc-226629-big-Data.db
Max: 03/01/2019 Min: 02/28/2019 Estimated droppable tombstones: 6.461017817804411E-9     
22G Mar 2 01:43 mc-227693-big-Data.db
Max: 03/02/2019 Min: 03/01/2019 Estimated droppable tombstones: 9.270087478134665E-10    
21G Mar 3 01:19 mc-228763-big-Data.db
Max: 03/03/2019 Min: 03/02/2019 Estimated droppable tombstones: 0.0       21G Mar 4 02:35
mc-229907-big-Data.db
Max: 03/04/2019 Min: 03/03/2019 Estimated droppable tombstones: 2.6800683107535888E-9    
22G Mar 5 01:50 mc-230915-big-Data.db
Max: 03/05/2019 Min: 03/04/2019 Estimated droppable tombstones: 2.703574931824122E-9     
22G Mar 6 00:35 mc-231983-big-Data.db
Max: 03/06/2019 Min: 03/05/2019 Estimated droppable tombstones: 0.0       22G Mar 7 00:38
mc-232951-big-Data.db
Max: 03/07/2019 Min: 03/06/2019 Estimated droppable tombstones: 0.0       11G Mar 7 12:17
mc-233426-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0       3.9M Mar 7 01:32
mc-233018-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0       427M Mar 7 12:42
mc-233459-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0       437M Mar 7 13:01
mc-233469-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0       455M Mar 7 12:52
mc-233464-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0       62M Mar 7 01:32
mc-233017-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0       681M Mar 7 12:17
mc-233443-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0       694M Mar 7 12:11
mc-233438-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0       795M Mar 7 12:34
mc-233454-big-Data.db
Max: 08/17/2018 Min: 08/17/2018 Estimated droppable tombstones: 4.202054543912095E-4     
141M Mar 6 21:17 mc-232845-big-Data.db
Max: 08/18/2018 Min: 08/17/2018 Estimated droppable tombstones: 0.052106761370398144     
1.1G Mar 6 21:08 mc-232827-big-Data.db
Max: 08/19/2018 Min: 08/18/2018 Estimated droppable tombstones: 0.0656319007897974       
1.1G Mar 6 21:15 mc-232838-big-Data.db
Max: 08/20/2018 Min: 08/19/2018 Estimated droppable tombstones: 0.12785613299472212      
1.1G Mar 6 21:05 mc-232823-big-Data.db
Max: 08/21/2018 Min: 08/20/2018 Estimated droppable tombstones: 0.11578795798812021      
1.1G Mar 6 21:09 mc-232829-big-Data.db
Max: 08/22/2018 Min: 08/21/2018 Estimated droppable tombstones: 0.11781698286628624      
1.1G Mar 6 21:06 mc-232824-big-Data.db
Max: 08/23/2018 Min: 08/22/2018 Estimated droppable tombstones: 0.015310752384131633     
1.1G Mar 6 21:11 mc-232832-big-Data.db
Max: 08/24/2018 Min: 08/23/2018 Estimated droppable tombstones: 6.239976748893364E-4     
1.1G Mar 6 21:14 mc-232837-big-Data.db
Max: 08/25/2018 Min: 08/24/2018 Estimated droppable tombstones: 0.007267002827070718     
1.1G Mar 6 21:16 mc-232842-big-Data.db
Max: 08/26/2018 Min: 08/25/2018 Estimated droppable tombstones: 0.015487525347912895     
1.1G Mar 6 21:12 mc-232834-big-Data.db
Max: 08/27/2018 Min: 08/26/2018 Estimated droppable tombstones: 0.22586925686932946      
1.2G Mar 5 03:53 mc-231079-big-Data.db
Max: 08/28/2018 Min: 08/27/2018 Estimated droppable tombstones: 0.20038331556336905      
1.2G Mar 5 08:11 mc-231268-big-Data.db
Max: 08/29/2018 Min: 08/28/2018 Estimated droppable tombstones: 0.2530757099874583       
1.3G Mar 4 19:01 mc-230693-big-Data.db
Max: 08/30/2018 Min: 08/29/2018 Estimated droppable tombstones: 0.2558579833614155       
1.3G Mar 5 17:23 mc-231701-big-Data.db
Max: 08/31/2018 Min: 08/30/2018 Estimated droppable tombstones: 0.252704679692973        
1.2G Mar 5 17:17 mc-231695-big-Data.db
Max: 09/01/2018 Min: 08/31/2018 Estimated droppable tombstones: 0.251179630123946        
1.2G Mar 5 15:59 mc-231633-big-Data.db
Max: 09/02/2018 Min: 09/01/2018 Estimated droppable tombstones: 0.2554852155936904       
1.2G Mar 5 16:29 mc-231658-big-Data.db
Max: 09/03/2018 Min: 09/02/2018 Estimated droppable tombstones: 0.25881950027198586      
1.2G Mar 5 16:00 mc-231635-big-Data.db
Max: 09/04/2018 Min: 09/03/2018 Estimated droppable tombstones: 0.7886645476349738       
4.7G Mar 4 20:41 mc-230737-big-Data.db
Max: 09/05/2018 Min: 09/04/2018 Estimated droppable tombstones: 0.8831512575108397       
13G Mar 5 03:13 mc-230996-big-Data.db
Max: 09/06/2018 Min: 09/05/2018 Estimated droppable tombstones: 0.881454703411404        
12G Mar 5 06:33 mc-231151-big-Data.db
Max: 09/07/2018 Min: 09/06/2018 Estimated droppable tombstones: 0.8771459061085191       
12G Mar 5 11:24 mc-231372-big-Data.db
Max: 09/08/2018 Min: 09/07/2018 Estimated droppable tombstones: 0.8753203300842707       
12G Mar 4 23:23 mc-230835-big-Data.db
Max: 09/09/2018 Min: 09/08/2018 Estimated droppable tombstones: 0.8808274303834315       
11G Mar 5 07:58 mc-231212-big-Data.db
Max: 09/10/2018 Min: 09/09/2018 Estimated droppable tombstones: 0.8809198437763183       
12G Mar 5 08:03 mc-231217-big-Data.db
Max: 09/11/2018 Min: 09/10/2018 Estimated droppable tombstones: 0.8723055307614438       
11G Mar 5 07:31 mc-231196-big-Data.db
Max: 09/12/2018 Min: 09/11/2018 Estimated droppable tombstones: 0.8747882208752266       
11G Mar 5 05:19 mc-231097-big-Data.db
Max: 09/13/2018 Min: 09/12/2018 Estimated droppable tombstones: 0.05224083315449286      
1.3G Mar 6 21:00 mc-232816-big-Data.db
Max: 09/14/2018 Min: 09/13/2018 Estimated droppable tombstones: 0.07163238619554278      
1.2G Mar 6 21:02 mc-232818-big-Data.db
Max: 09/15/2018 Min: 09/14/2018 Estimated droppable tombstones: 0.10299448148502635      
1.2G Mar 6 21:03 mc-232820-big-Data.db
Max: 09/16/2018 Min: 09/15/2018 Estimated droppable tombstones: 0.12947413624859622      
1.2G Mar 1 14:51 mc-227283-big-Data.db
Max: 09/17/2018 Min: 09/16/2018 Estimated droppable tombstones: 0.14577734759443223      
1.2G Mar 2 14:55 mc-228346-big-Data.db
Max: 09/18/2018 Min: 09/17/2018 Estimated droppable tombstones: 0.15627725735914008      
1.2G Mar 3 12:35 mc-229331-big-Data.db
Max: 09/19/2018 Min: 09/18/2018 Estimated droppable tombstones: 0.19208129168867066      
1.3G Mar 4 10:23 mc-230295-big-Data.db
Max: 09/20/2018 Min: 09/19/2018 Estimated droppable tombstones: 0.249042970270552        
1.4G Mar 5 10:21 mc-231370-big-Data.db
Max: 09/21/2018 Min: 09/20/2018 Estimated droppable tombstones: 0.2923327861443509       
1.4G Mar 5 10:39 mc-231385-big-Data.db
Max: 09/22/2018 Min: 09/21/2018 Estimated droppable tombstones: 0.4694543993137898       
1.7G Mar 5 07:25 mc-231230-big-Data.db
Max: 09/23/2018 Min: 09/22/2018 Estimated droppable tombstones: 0.8812610467316245       
7.0G Mar 5 14:31 mc-231532-big-Data.db
Max: 09/24/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8963227271743491       
9.6G Mar 4 18:56 mc-230653-big-Data.db
Max: 09/25/2018 Min: 09/24/2018 Estimated droppable tombstones: 0.8828679015527205       
13G Mar 4 22:07 mc-230775-big-Data.db
Max: 09/26/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8706920830020076       
14G Mar 5 07:01 mc-231162-big-Data.db
Max: 09/27/2018 Min: 09/26/2018 Estimated droppable tombstones: 0.8774074013483522       
14G Mar 5 13:03 mc-231437-big-Data.db
Max: 09/28/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8446102849522781       
14G Mar 5 00:43 mc-230864-big-Data.db
Max: 09/29/2018 Min: 09/28/2018 Estimated droppable tombstones: 0.8808040881298336       
12G Mar 5 05:54 mc-231117-big-Data.db
Max: 09/30/2018 Min: 09/29/2018 Estimated droppable tombstones: 0.88253551363474         
12G Mar 5 03:46 mc-231028-big-Data.db
Max: 10/01/2018 Min: 09/24/2018 Estimated droppable tombstones: 0.8788732337462403       
13G Mar 5 12:12 mc-231403-big-Data.db
Max: 10/02/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.883335202175054        
12G Mar 5 00:29 mc-230874-big-Data.db
Max: 10/03/2018 Min: 10/02/2018 Estimated droppable tombstones: 0.8771751169697322       
12G Mar 5 17:47 mc-231677-big-Data.db
Max: 10/04/2018 Min: 10/03/2018 Estimated droppable tombstones: 0.8822618238791344       
13G Mar 5 19:33 mc-231737-big-Data.db
Max: 10/05/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8865947931678021       
13G Mar 4 20:45 mc-230717-big-Data.db
Max: 10/06/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8800199984853445       
13G Mar 5 03:32 mc-231010-big-Data.db
Max: 10/07/2018 Min: 10/06/2018 Estimated droppable tombstones: 0.8912633931978029       
13G Mar 5 14:36 mc-231519-big-Data.db
Max: 10/08/2018 Min: 10/07/2018 Estimated droppable tombstones: 0.8776893254692907       
13G Mar 5 01:58 mc-230939-big-Data.db
Max: 10/09/2018 Min: 10/08/2018 Estimated droppable tombstones: 0.8856079371268669       
13G Mar 5 14:08 mc-231496-big-Data.db
Max: 10/10/2018 Min: 10/09/2018 Estimated droppable tombstones: 0.8779972870791256       
13G Mar 5 16:48 mc-231621-big-Data.db
Max: 10/11/2018 Min: 10/10/2018 Estimated droppable tombstones: 0.8798199877431634       
13G Mar 5 00:22 mc-230863-big-Data.db
Max: 10/12/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8818200837221603       
13G Mar 5 11:07 mc-231347-big-Data.db
Max: 10/13/2018 Min: 10/12/2018 Estimated droppable tombstones: 0.8792880344026048       
13G Mar 4 20:01 mc-230689-big-Data.db
Max: 10/14/2018 Min: 10/13/2018 Estimated droppable tombstones: 0.8893721720823108       
13G Mar 4 21:58 mc-230770-big-Data.db
Max: 10/15/2018 Min: 10/14/2018 Estimated droppable tombstones: 0.8866647009668137       
13G Mar 5 01:58 mc-230933-big-Data.db
Max: 10/16/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8800707952691067       
14G Mar 5 11:00 mc-231343-big-Data.db
Max: 10/17/2018 Min: 09/19/2018 Estimated droppable tombstones: 0.8810867844724584       
15G Mar 5 05:00 mc-231072-big-Data.db
Max: 10/18/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8785127729091343       
15G Mar 5 09:07 mc-231256-big-Data.db
Max: 10/19/2018 Min: 09/20/2018 Estimated droppable tombstones: 0.8876021364430524       
15G Mar 5 04:42 mc-231060-big-Data.db
Max: 10/20/2018 Min: 10/19/2018 Estimated droppable tombstones: 0.881872857492371        
13G Mar 5 17:58 mc-231682-big-Data.db
Max: 10/21/2018 Min: 10/20/2018 Estimated droppable tombstones: 0.8848086228545486       
13G Mar 5 13:51 mc-231483-big-Data.db
Max: 10/22/2018 Min: 08/27/2018 Estimated droppable tombstones: 0.8792001876878928       
15G Mar 5 15:24 mc-231552-big-Data.db
Max: 10/23/2018 Min: 08/27/2018 Estimated droppable tombstones: 0.886772503866838        
14G Mar 5 00:37 mc-230872-big-Data.db
Max: 10/24/2018 Min: 10/23/2018 Estimated droppable tombstones: 0.883097444361784        
13G Mar 4 21:55 mc-230766-big-Data.db
Max: 10/25/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8725515297254708       
14G Mar 5 03:38 mc-231015-big-Data.db
Max: 10/26/2018 Min: 10/25/2018 Estimated droppable tombstones: 0.8818153097020685       
12G Mar 5 02:17 mc-230953-big-Data.db
Max: 10/27/2018 Min: 10/26/2018 Estimated droppable tombstones: 0.8848182273248961       
12G Mar 5 01:12 mc-230905-big-Data.db
Max: 10/28/2018 Min: 10/27/2018 Estimated droppable tombstones: 0.8863910139910516       
12G Mar 5 15:16 mc-231554-big-Data.db
Max: 10/29/2018 Min: 10/28/2018 Estimated droppable tombstones: 0.8804707550662745       
13G Mar 5 19:53 mc-231768-big-Data.db
Max: 10/30/2018 Min: 10/29/2018 Estimated droppable tombstones: 0.8848917993137158       
13G Mar 5 17:06 mc-231631-big-Data.db
Max: 10/31/2018 Min: 09/04/2018 Estimated droppable tombstones: 0.8744904945244535       
15G Mar 5 16:01 mc-231581-big-Data.db
Max: 11/01/2018 Min: 10/31/2018 Estimated droppable tombstones: 0.8844023760442303       
13G Mar 5 16:56 mc-231623-big-Data.db
Max: 11/02/2018 Min: 09/06/2018 Estimated droppable tombstones: 0.876832422513591        
15G Mar 5 17:42 mc-231663-big-Data.db
Max: 11/03/2018 Min: 11/02/2018 Estimated droppable tombstones: 0.8884698595625823       
13G Mar 5 12:00 mc-231393-big-Data.db
Max: 11/04/2018 Min: 11/03/2018 Estimated droppable tombstones: 0.8856117904594102       
13G Mar 5 05:03 mc-231078-big-Data.db
Max: 11/05/2018 Min: 11/04/2018 Estimated droppable tombstones: 0.8850104959605515       
13G Mar 5 06:28 mc-231131-big-Data.db
Max: 11/06/2018 Min: 11/05/2018 Estimated droppable tombstones: 0.8876672835148812       
13G Mar 4 23:03 mc-230811-big-Data.db
Max: 11/07/2018 Min: 11/06/2018 Estimated droppable tombstones: 0.8819559232651852       
13G Mar 5 19:52 mc-231763-big-Data.db
Max: 11/08/2018 Min: 11/07/2018 Estimated droppable tombstones: 0.8890240896109401       
14G Mar 5 09:34 mc-231277-big-Data.db
Max: 11/09/2018 Min: 09/13/2018 Estimated droppable tombstones: 0.8889195336887382       
16G Mar 5 14:22 mc-231498-big-Data.db
Max: 11/10/2018 Min: 11/09/2018 Estimated droppable tombstones: 0.8901968237826773       
14G Mar 5 13:12 mc-231440-big-Data.db
Max: 11/11/2018 Min: 11/10/2018 Estimated droppable tombstones: 0.8823935870159266       
14G Mar 5 12:29 mc-231407-big-Data.db
Max: 11/12/2018 Min: 11/11/2018 Estimated droppable tombstones: 0.8821515266485579       
14G Mar 5 07:09 mc-231164-big-Data.db
Max: 11/13/2018 Min: 09/17/2018 Estimated droppable tombstones: 0.8821863572044562       
16G Mar 5 17:27 mc-231638-big-Data.db
Max: 11/14/2018 Min: 11/13/2018 Estimated droppable tombstones: 0.8837352268527888       
14G Mar 5 19:24 mc-231730-big-Data.db
Max: 11/15/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8790743796626552       
15G Mar 5 09:27 mc-231264-big-Data.db
Max: 11/16/2018 Min: 09/23/2018 Estimated droppable tombstones: 0.8844277571061905       
15G Mar 5 16:35 mc-231607-big-Data.db
Max: 11/17/2018 Min: 11/16/2018 Estimated droppable tombstones: 0.8862673108309931       
14G Mar 5 08:03 mc-231204-big-Data.db
Max: 11/18/2018 Min: 11/17/2018 Estimated droppable tombstones: 0.8842595514003214       
14G Mar 5 18:52 mc-231717-big-Data.db
Max: 11/19/2018 Min: 09/24/2018 Estimated droppable tombstones: 0.881639236418788        
15G Mar 4 23:34 mc-230826-big-Data.db
Max: 11/20/2018 Min: 11/19/2018 Estimated droppable tombstones: 0.886075536820344        
14G Mar 4 23:00 mc-230805-big-Data.db
Max: 11/21/2018 Min: 11/20/2018 Estimated droppable tombstones: 0.889928801493522        
14G Mar 5 09:35 mc-231279-big-Data.db
Max: 11/22/2018 Min: 11/21/2018 Estimated droppable tombstones: 0.8861353317060062       
14G Mar 5 20:00 mc-231776-big-Data.db
Max: 11/23/2018 Min: 11/22/2018 Estimated droppable tombstones: 0.8829576920997242       
14G Mar 5 07:49 mc-231198-big-Data.db
Max: 11/24/2018 Min: 11/23/2018 Estimated droppable tombstones: 0.8821505816554567       
13G Mar 5 03:25 mc-230998-big-Data.db
Max: 11/25/2018 Min: 11/24/2018 Estimated droppable tombstones: 0.880930424220212        
13G Mar 4 20:17 mc-230696-big-Data.db
Max: 11/26/2018 Min: 11/25/2018 Estimated droppable tombstones: 0.8820117087435542       
13G Mar 4 21:32 mc-230743-big-Data.db
Max: 11/27/2018 Min: 11/26/2018 Estimated droppable tombstones: 0.8791004516800335       
13G Mar 5 03:23 mc-230997-big-Data.db
Max: 11/28/2018 Min: 11/27/2018 Estimated droppable tombstones: 0.8788965149724617       
13G Mar 5 01:52 mc-230929-big-Data.db
Max: 11/29/2018 Min: 11/28/2018 Estimated droppable tombstones: 0.8877500310616938       
13G Mar 4 22:50 mc-230803-big-Data.db
Max: 11/30/2018 Min: 11/29/2018 Estimated droppable tombstones: 0.8754630223343176       
13G Mar 4 21:26 mc-230742-big-Data.db
Max: 12/01/2018 Min: 11/30/2018 Estimated droppable tombstones: 0.8762374681745563       
12G Mar 4 21:34 mc-230755-big-Data.db
Max: 12/02/2018 Min: 12/01/2018 Estimated droppable tombstones: 0.8787902962561704       
12G Mar 5 04:31 mc-231059-big-Data.db
Max: 12/03/2018 Min: 12/02/2018 Estimated droppable tombstones: 0.8853087223736261       
12G Mar 5 15:16 mc-231551-big-Data.db
Max: 12/04/2018 Min: 12/03/2018 Estimated droppable tombstones: 0.8803649205154546       
12G Mar 5 10:06 mc-231309-big-Data.db
Max: 12/05/2018 Min: 12/04/2018 Estimated droppable tombstones: 0.8824805012470633       
12G Mar 5 10:33 mc-231334-big-Data.db
Max: 12/06/2018 Min: 12/05/2018 Estimated droppable tombstones: 0.7605552563033167       
4.1G Mar 5 08:12 mc-231253-big-Data.db
Max: 12/07/2018 Min: 12/06/2018 Estimated droppable tombstones: 0.7748787995519647       
3.9G Mar 5 10:55 mc-231386-big-Data.db
Max: 12/08/2018 Min: 12/07/2018 Estimated droppable tombstones: 0.7998981602050579       
4.1G Mar 5 08:37 mc-231275-big-Data.db
Max: 12/09/2018 Min: 12/08/2018 Estimated droppable tombstones: 0.8047662079574316       
4.5G Mar 5 03:35 mc-231043-big-Data.db
Max: 12/10/2018 Min: 12/09/2018 Estimated droppable tombstones: 0.7987046261073453       
4.8G Mar 4 23:36 mc-230870-big-Data.db
Max: 12/11/2018 Min: 12/10/2018 Estimated droppable tombstones: 0.8346316850246404       
5.6G Mar 5 13:10 mc-231478-big-Data.db
Max: 12/12/2018 Min: 12/11/2018 Estimated droppable tombstones: 0.8336216107728608       
6.1G Mar 5 00:06 mc-230888-big-Data.db
Max: 12/13/2018 Min: 12/12/2018 Estimated droppable tombstones: 0.8566337089121          
7.2G Mar 5 02:46 mc-230993-big-Data.db
Max: 12/14/2018 Min: 12/13/2018 Estimated droppable tombstones: 0.8137644691768783       
4.7G Mar 5 10:32 mc-231358-big-Data.db
Max: 12/15/2018 Min: 12/14/2018 Estimated droppable tombstones: 0.8166609937509232       
4.6G Mar 5 13:59 mc-231525-big-Data.db
Max: 12/16/2018 Min: 12/15/2018 Estimated droppable tombstones: 0.8085604043211527       
4.8G Mar 5 05:00 mc-231110-big-Data.db
Max: 12/17/2018 Min: 12/16/2018 Estimated droppable tombstones: 0.8124008277006111       
5.0G Mar 4 20:34 mc-230739-big-Data.db
Max: 12/18/2018 Min: 12/17/2018 Estimated droppable tombstones: 0.8197544452946743       
5.0G Mar 5 12:03 mc-231430-big-Data.db
Max: 12/19/2018 Min: 12/18/2018 Estimated droppable tombstones: 0.7604684134873694       
5.7G Mar 4 21:08 mc-230768-big-Data.db
Max: 12/20/2018 Min: 12/19/2018 Estimated droppable tombstones: 0.6276716162431576       
6.8G Mar 4 22:39 mc-230832-big-Data.db
Max: 12/21/2018 Min: 12/20/2018 Estimated droppable tombstones: 0.6262830796548643       
6.9G Mar 4 21:23 mc-230778-big-Data.db
Max: 12/22/2018 Min: 12/21/2018 Estimated droppable tombstones: 0.6245678218315354       
6.7G Mar 5 09:22 mc-231304-big-Data.db
Max: 12/23/2018 Min: 12/22/2018 Estimated droppable tombstones: 0.6339901894339154       
6.7G Mar 5 00:06 mc-230897-big-Data.db
Max: 12/24/2018 Min: 12/23/2018 Estimated droppable tombstones: 0.6401085489180292       
6.8G Mar 5 00:17 mc-230901-big-Data.db
Max: 12/25/2018 Min: 12/24/2018 Estimated droppable tombstones: 0.648027924752315        
6.9G Mar 4 22:04 mc-230809-big-Data.db
Max: 12/26/2018 Min: 12/25/2018 Estimated droppable tombstones: 0.6465660696516876       
7.0G Mar 4 23:16 mc-230856-big-Data.db
Max: 12/27/2018 Min: 12/26/2018 Estimated droppable tombstones: 0.5464676457788102       
5.9G Mar 5 08:46 mc-231285-big-Data.db
Max: 12/28/2018 Min: 12/27/2018 Estimated droppable tombstones: 0.5556336150105652       
5.8G Mar 5 09:03 mc-231298-big-Data.db
Max: 12/29/2018 Min: 12/28/2018 Estimated droppable tombstones: 0.5884672237873865       
6.1G Mar 4 20:32 mc-230741-big-Data.db
Max: 12/30/2018 Min: 12/29/2018 Estimated droppable tombstones: 0.6116207911770754       
6.3G Mar 4 21:52 mc-230801-big-Data.db
Max: 12/31/2018 Min: 12/30/2018 Estimated droppable tombstones: 0.6156449592384619       
6.6G Mar 5 09:48 mc-231332-big-Data.db



Currently our data on disk is filling up quickly because we are unable to successfully evict
this data. Is there a way to


  1.

1. Cleanup what is currently taking up so much disk space

  2.

2. Mitigate this entirely in the future


Any help would be greatly appreciated!!

Thanks,

Nick Hatfield

From: Surbhi Gupta <surbhi.gupta01@gmail.com<mailto:surbhi.gupta01@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Thursday, March 7, 2019 at 11:50 AM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: can i...

Send the details

On Thu, Mar 7, 2019 at 8:45 AM Nick Hatfield <nick.hatfield@metricly.com<mailto:nick.hatfield@metricly.com>>
wrote:
Use this email to get some insight on how to fix database issues in our cluster?

Mime
View raw message