cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: Anticompaction causing significant increase in disk usage
Date Wed, 12 Sep 2018 11:30:56 GMT
Hello Martin,

How do you perform the repairs?

Are you using incremental repairs or full repairs but without subranges?
Alex described issues related to these repairs here:
http://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html
.

*tl;dr: *

The only way to perform repair without anticompaction in “modern” versions
> of Apache Cassandra is subrange repair, which fully skips anticompaction.
> To perform a subrange repair correctly, you have three options :
> - Compute valid token subranges yourself and script repairs accordingly
> - Use the Cassandra range repair script which performs subrange repair
> - Use Cassandra Reaper, which also performs subrange repair


If you can prevent anti-compaction, disk space growth should be more
predictable.

There might be more solutions now out there, C* should also soon be shipped
with a side-car it's being actively discussed. Finally, Incremental repairs
will receive important fixes in Cassandra 4.0, Alex also wrote about this
too (yes, this guy loves repairs ¯\_(ツ)_/¯)
http://thelastpickle.com/blog/2018/09/10/incremental-repair-improvements-in-cassandra-4.html

I believe (and hope) this information is relevant to help you fix this
issue.

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Le mer. 12 sept. 2018 à 10:14, Martin Mačura <m.macura@gmail.com> a écrit :

> Hi,
> we're on cassandra 3.11.2 . During an anticompaction after repair,
> TotalDiskSpaceUsed value of one table gradually went from 700GB to
> 1180GB, and then suddenly jumped back to 700GB.  This happened on all
> nodes involved in the repair. There was no change in PercentRepaired
> during or after this process. SSTable count is currently 857, with a
> peak of 2460 during the repair.
>
> Table is using TWCS with 1-day time window.  Most daily SSTables are
> around 1 GB but the oldest one is 156 GB - caused by a major
> compaction.
>
> system.log.6.zip:INFO  [CompactionExecutor:9923] 2018-09-10
> 15:29:54,238 CompactionManager.java:649 - [repair
> #88c36e30-b4cb-11e8-bebe-cd3efd73ed33] Starting anticompaction for ...
> on 519 [...]  SSTables
> ...
> system.log:INFO  [CompactionExecutor:9923] 2018-09-12 00:29:39,262
> CompactionManager.java:1524 - Anticompaction completed successfully,
> anticompacted from 0 to 518 sstable(s).
>
> What could be the cause of the temporary increase, and how can we
> prevent it?  We are concerned about running out of disk space soon.
>
> Thanks for any help
>
> Martin
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>

Mime
View raw message