cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Brotman" <kenbrot...@yahoo.com.INVALID>
Subject RE: Urgent Problem - Disk full
Date Wed, 04 Apr 2018 18:18:55 GMT
Agreed that you tend to add capacity to nodes or add nodes once you know you have no unneeded
data in the cluster.

 

From: Alain RODRIGUEZ [mailto:arodrime@gmail.com] 
Sent: Wednesday, April 04, 2018 9:10 AM
To: user cassandra.apache.org
Subject: Re: Urgent Problem - Disk full

 

Hi,

 

When the disks are full, here are the options I can think of depending on the situation and
how 'full' the disk really is:

- Add capacity - Add a disk, use JBOD adding a second location folder for the sstables and
move some of them around then restart Cassandra. Or add a new node.
- Reduce disk space used. Some options come to my mind to reduce space used:


1 - Clean tombstones if any (use sstablemetadata for example to check the number of tombstones).
If you have some not being purged, my first guess would be to set 'unchecked_tombstone_compaction'
to 'true' at the node level. Yet be aware that this will trigger some compactions, that before
freeing space, start by taking some more temporary!

 

If remaining space is really low on one node, you can control to compact only on the sstables
having the higher tombstone ratio after you made the change above and that fit in the disk
space you have left. It can even be scripted. It worked for me in the past with disk 100%
full. If you do so, you might have to disable/reenable automatic compactions at key moments
as well

 

2 -  If you added nodes recently to the data center you can consider running a 'nodetool cleanup',
but here again, it will start by using more space for temporary sstables, and might have no
positive impacts if the node only own data for its token ranges.

 

3 - Another common way to easily claim space is to clear snapshots that are not needed and
might have been forgotten or taken by Cassandra: 'nodetool clearsnapshot'. This has no other
risk than removing a useful backup.


4 - Delete data from this table or another table (effectively), directly removing the sstables
indeed - as you use TWCS. If you don't need the data anyway.

 

5 - Truncate one of those other tables we tend to have that are written 'just in case' and
actually never used and never read for months. It has been a powerful way out of this situation
for me in the past too :). I would say: be sure that the disk space is used properly.

 

 

There is zero reason to believe a full repair would make this better and a lot of reason to
believe it’ll make it worse

 

I second that too, just in case. Really, do not run a repair. The only thing it could do is
bring more data to a node that really don't need it for now.

 

Finally, when this is behind you, the disk size is something you could consider monitoring
as it is way easier to fix it when the disk is not completely full and it can be fixed preemptively.
Usually, 50 to 20% of free disk is recommended depending on your use case.

 

C*heers,

-----------------------

Alain Rodriguez - @arodream - alain@thelastpickle.com

France / Spain

 

The Last Pickle - Apache Cassandra Consulting

http://www.thelastpickle.com <http://wwwthelastpickle.com> 

 

2018-04-04 15:34 GMT+01:00 Kenneth Brotman <kenbrotman@yahoo.com.invalid>:

There's also the old snapshots to remove that could be a significant amount of memory.


-----Original Message-----
From: Kenneth Brotman [mailto:kenbrotman@yahoo.com.INVALID]
Sent: Wednesday, April 04, 2018 7:28 AM
To: user@cassandra.apache.org
Subject: RE: Urgent Problem - Disk full

Jeff,

Just wondering: why wouldn't the answer be to:
        1. move anything you want to archive to colder storage off the cluster,
        2. nodetool cleanup
        3. snapshot
        4. use delete command to remove archived data.

Kenneth Brotman

-----Original Message-----
From: Jeff Jirsa [mailto:jjirsa@gmail.com]
Sent: Wednesday, April 04, 2018 7:10 AM
To: user@cassandra.apache.org
Subject: Re: Urgent Problem - Disk full

Yes, this works in TWCS.

Note though that if you have tombstone compaction subproperties set, there may be sstables
with newer filesystem timestamps that actually hold older Cassandra data, in which case sstablemetadata
can help finding the sstables with truly old timestamps

Also if you’ve expanded the cluster over time and you see an imbalance of disk usage on
the oldest hosts, “nodetool cleanup” will likely free up some of that data



--
Jeff Jirsa


> On Apr 4, 2018, at 4:32 AM, Jürgen Albersdorfer <Juergen.Albersdorfer@zweiradteile.net>
wrote:
>
> Hi,
>
> I have an urgent Problem. - I will run out of disk space in near future.
> Largest Table is a Time-Series Table with TimeWindowCompactionStrategy (TWCS) and default_time_to_live
= 0
> Keyspace Replication Factor RF=3. I run C* Version 3.11.2
> We have grown the Cluster over time, so SSTable files have different Dates on different
Nodes.
>
> From Application Standpoint it would be safe to loose some of the oldest Data.
>
> Is it safe to delete some of the oldest SSTable Files, which will no longer get touched
by TWCS Compaction any more, while Node is clean Shutdown? - And doing so for one Node after
another?
>
> Or maybe there is a different way to free some disk space? - Any suggestions?
>
> best regards
> Jürgen Albersdorfer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org <mailto:user-help@cassandraapache.org>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <mailto:user-unsubscribe@cassandra.apacheorg>

For additional commands, e-mail: user-help@cassandra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <mailto:user-unsubscribe@cassandra.apacheorg>

For additional commands, e-mail: user-help@cassandra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <mailto:user-unsubscribe@cassandra.apacheorg>

For additional commands, e-mail: user-help@cassandra.apache.org

 


Mime
View raw message