cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steinmaurer, Thomas" <thomas.steinmau...@dynatrace.com>
Subject RE: Massive deletes -> major compaction?
Date Fri, 22 Sep 2017 12:05:56 GMT
Additional to Kurt’s reply. Double disk usage is really the worst case. Most of the time
you are fine having > largest column family free disk available.

Also take local snapshots into account. Even after a finished major compaction, disk space
may have not been reclaimed, if snapshot sym links still keep disk usage of already compacted
SSTable alive.

Regards,
Thomas

From: Durity, Sean R [mailto:SEAN_R_DURITY@homedepot.com]
Sent: Freitag, 22. September 2017 13:38
To: user@cassandra.apache.org
Subject: RE: Massive deletes -> major compaction?

Thanks for the pointer. I had never heard of this. While it seems that it could help, I think
our rules for determining which records to keep are not supported. Also, this requires adding
a new jar to production. Too risky at this point.


Sean Durity

From: Jon Haddad [mailto:jonathan.haddad@gmail.com] On Behalf Of Jon Haddad
Sent: Thursday, September 21, 2017 2:59 PM
To: user <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: Massive deletes -> major compaction?

Have you considered the fantastic DeletingCompactionStrategy?  https://github.com/protectwise/cassandra-util/tree/master/deleting-compaction-strategy<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_protectwise_cassandra-2Dutil_tree_master_deleting-2Dcompaction-2Dstrategy&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=XbpSdVHHLZeNv3mp3UkL122S3UryXjaG-ROk8SK6Oro&s=rP7k5CqnOsEASTayoqmU-BOCfo-R0tqg6VGBc3sSXoE&e=>


On Sep 21, 2017, at 11:51 AM, Jeff Jirsa <jjirsa@gmail.com<mailto:jjirsa@gmail.com>>
wrote:

The major compaction is most efficient but can temporarily double (nearly) disk usage - if
you can afford that, go for it.

Alternatively you can do a user-defined compaction on each sstable in reverse generational
order (oldest first) and as long as the data is minimally overlapping it’ll purge tombstones
that way as well - takes longer but much less disk involved.


--
Jeff Jirsa


On Sep 21, 2017, at 11:27 AM, Durity, Sean R <SEAN_R_DURITY@homedepot.com<mailto:SEAN_R_DURITY@homedepot.com>>
wrote:
Cassandra version 2.0.17 (yes, it’s old – waiting for new hardware/new OS to upgrade)

In a long-running system with billions of rows, TTL was not set. So a one-time purge is being
planned to reduce disk usage. Records older than a certain date will be deleted. The table
uses size-tiered compaction. Deletes are probably 25-40% of the complete data set. To actually
recover the disk space, would you recommend a major compaction after the gc_grace_seconds
time? I expect compaction would then need to be scheduled regularly (ick)…

We also plan to re-insert the remaining data with a calculated TTL, which could also benefit
from compaction.


Sean Durity

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is
intended solely for the addressee. Access to this Email by anyone else is unauthorized. If
you are not the intended recipient, any disclosure, copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject to the terms and
conditions expressed in any applicable governing The Home Depot terms of business or client
engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy
and content of this attachment and for any damages or losses arising from any inaccuracies,
errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature,
which may be contained in this attachment and shall not be liable for direct, indirect, consequential
or special damages in connection with this e-mail message or its attachment.


________________________________

The information in this Internet Email is confidential and may be legally privileged. It is
intended solely for the addressee. Access to this Email by anyone else is unauthorized. If
you are not the intended recipient, any disclosure, copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject to the terms and
conditions expressed in any applicable governing The Home Depot terms of business or client
engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy
and content of this attachment and for any damages or losses arising from any inaccuracies,
errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature,
which may be contained in this attachment and shall not be liable for direct, indirect, consequential
or special damages in connection with this e-mail message or its attachment.
The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
Mime
View raw message