cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-7066) Simplify (and unify) cleanup of compaction leftovers
Date Thu, 30 Jul 2015 10:52:05 GMT


Stefania commented on CASSANDRA-7066:

[~benedict] thank you for your comments. I've updated the description and implemented the
incremental checksum checks. You may want to review all together later or take a quick look
at what we do when a check fails, which is not much, just leave the files there and log an
error, not sure what a panic error is exactly?

bq. Perhaps we should offer a tool, like sstablelister, to offline complete any transaction
log files, which can be specified as prep-work for backup restore

sstablelister will return all tx logs as well as temporary files: {{sstablelister -o -t tmp
ks table}}, so they can delete tx logs and tmp files with one command, which is equivalent
to offline complete.

I still need clarifications regarding the last point:

bq. I agree about individual files over-complicating things, however we _could _ perhaps take
the maximum last modified of all files with that descriptor, and then delete the files in
ascending order of last modified. This would just help the log files be robust to backups
being copied in with these files still present and referencing them as old, and also to some
potential effects of problems like CASSANDRA-9908.

Here we are talking about old files only, right? Because new files haven't even been created
yet when trackNew() is called and it must be like so to ensure we don't have any window where
physical temporary files exist that are not tracked. So we would take the last modified of
all files with that descriptor when obsoleted is called and log this time along with the descriptor?
Then when it's time to delete old files we refuse to delete any more recent files, do I understand
this correctly?

[~nickmbailey] your points are well noted:

bq. The snapshot command should create a full backup of a keyspace/table on the node. The
directories created from the snapshot should be all that is required to restore that keyspace/table
on that node to the point in time that the snapshot was taken.

The snapshot command creates hard links to the sstable files, it won't look at transaction
log files. The new lifecycle transaction ensures the sstables live for a cfs are valid and
not temporary.

bq. A snapshot should be restorable either via the sstableloader tool or by manually copying
the files from the snapshot in to place (given the same schema/topology). If copying the files
into place manually, restarting the node or making an additional call to load the sstables
may be required.

If copying the files manually users will have to remove any partial transaction log files
and their temporary files. This can be done via sstablelister with one command, for example
{{for x in $(sstablelister -o -t tmp ks table); do rm $x; done)}}. sstable loader is implemented
via streaming so the uploaded sstables will automatically get a new generation and won't clash
with older sstables involved in transactions.

bq. When using the sstableloader tool I should be able to restore data taken from a snapshot
regardless of what data exists on the node or is currently being written.

As per above, this should be the case.

> Simplify (and unify) cleanup of compaction leftovers
> ----------------------------------------------------
>                 Key: CASSANDRA-7066
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Benedict
>            Assignee: Stefania
>            Priority: Minor
>              Labels: benedict-to-commit, compaction
>             Fix For: 3.0 alpha 1
>         Attachments: 7066.txt
> Currently we manage a list of in-progress compactions in a system table, which we use
to cleanup incomplete compactions when we're done. The problem with this is that 1) it's a
bit clunky (and leaves us in positions where we can unnecessarily cleanup completed files,
or conversely not cleanup files that have been superceded); and 2) it's only used for a regular
compaction - no other compaction types are guarded in the same way, so can result in duplication
if we fail before deleting the replacements.
> I'd like to see each sstable store in its metadata its direct ancestors, and on startup
we simply delete any sstables that occur in the union of all ancestor sets. This way as soon
as we finish writing we're capable of cleaning up any leftovers, so we never get duplication.
It's also much easier to reason about.

This message was sent by Atlassian JIRA

View raw message