cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nimi Wariboko Jr <n...@channelmeter.com>
Subject Dealing with Corrupted/Missing LCS Metadata
Date Tue, 23 Aug 2016 22:21:37 GMT
Hi,

I'm trying to solve an issue we are seeing on our production cluster, in
various LCS tables, with Cassandra 3.5. I believe it is related
to CASSANDRA-11373 (however CASSANDRA-11373 is marked as resolved in 3.5).

I'm not sure what causes it, but eventually `getCandidatesFor` in
LeveledManifest.java (
https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java#L553)
gets stuck in an infinite loop on L0 (and eventually goes into a GC
thrashing loop). Investigating further, it looks like `getCandidatesFor`
returns a mix of sstables, some of which don't exist on disk (same symptom
as 11373).

What I'd like to know is what I can do here? For smaller tables we switched
to STS, but we have a very large LCS timeseries (~300GB per node) table,
that would take ages to switch over and really affect our performance on
STS (offline scrub doesn't work either).

Would it suffice to add a check here to ensure the file exists and evicting
the file if it doesn't? Or is there anyway I could force the rewrite of the
table's metadata?

Nimi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message