cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Coli <rc...@eventbrite.com>
Subject Re: Compaction issues, 2.0.12
Date Mon, 06 Jul 2015 20:51:53 GMT
On Mon, Jul 6, 2015 at 1:46 PM, Jeff Williams <jeffw@wherethebitsroam.com>
wrote:

> 1) Cassandra version is 2.0.12.
>


> 2) Interesting. Looking at JMX org.apache.cassandra.db -> ColumnFamilies
> -> trackcontent -> track_content -> Attributes, I get:
>
> LiveDiskSpaceUsed: 17788740448, i.e. ~17GB
> LiveSSTableCount: 3
> TotalDiskSpaceUsed: 55714084629, i.e. ~55GB
>
> So it obviously knows about the extra disk space even though the "live"
> space looks correct. I couldn't find anything to identify the actual files
> though.
>

That's what I would expect.


> 3) So that was even more interesting. After restarting the cassandra
> daemon, the sstables were not deleted and now the same JMX attributes are:
>
> LiveDiskSpaceUsed: 55681040579, i.e. ~55GB
> LiveSSTableCount: 8
> TotalDiskSpaceUsed: 55681040579, i.e. ~55GB
>
> So some of my non-live tables are back live again, and obviously some of
> the big ones!!
>

This is permanently fatal to consistency; sorry if I was not clear enough
that if they were not live, there was some risk of Cassandra considering
them live again upon restart.

If I were you, I would either stop the node and remove the files you know
shouldn't be live or do a major compaction ASAP.

The behavior you just encountered sounds like a bug, and it is a rather
serious one. SSTables which should be dead being marked live is very bad
for consistency.

Do you see any exceptions in your logs or anything? If you can repro, you
should file a JIRA ticket with the apache project...

=Rob

Mime
View raw message