cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Ferland <...@tubularlabs.com>
Subject Re: Compaction issues, 2.0.12
Date Mon, 06 Jul 2015 21:46:41 GMT
I’ve seen the same thing: https://issues.apache.org/jira/browse/CASSANDRA-9577 <https://issues.apache.org/jira/browse/CASSANDRA-9577>

I’ve had cases where a restart clears the old tables, and I’ve had cases where a restart
considers the old tables to be live.

> On Jul 6, 2015, at 1:51 PM, Robert Coli <rcoli@eventbrite.com> wrote:
> 
> On Mon, Jul 6, 2015 at 1:46 PM, Jeff Williams <jeffw@wherethebitsroam.com <mailto:jeffw@wherethebitsroam.com>>
wrote:
> 1) Cassandra version is 2.0.12. 
>  
> 2) Interesting. Looking at JMX org.apache.cassandra.db -> ColumnFamilies -> trackcontent
-> track_content -> Attributes, I get:
> 
> LiveDiskSpaceUsed: 17788740448 <tel:17788740448>, i.e. ~17GB
> LiveSSTableCount: 3
> TotalDiskSpaceUsed: 55714084629, i.e. ~55GB
> 
> So it obviously knows about the extra disk space even though the "live" space looks correct.
I couldn't find anything to identify the actual files though.
> 
> That's what I would expect.
>  
> 3) So that was even more interesting. After restarting the cassandra daemon, the sstables
were not deleted and now the same JMX attributes are:
> 
> LiveDiskSpaceUsed: 55681040579, i.e. ~55GB
> LiveSSTableCount: 8
> TotalDiskSpaceUsed: 55681040579, i.e. ~55GB
> 
> So some of my non-live tables are back live again, and obviously some of the big ones!!
> 
> This is permanently fatal to consistency; sorry if I was not clear enough that if they
were not live, there was some risk of Cassandra considering them live again upon restart.
> 
> If I were you, I would either stop the node and remove the files you know shouldn't be
live or do a major compaction ASAP. 
> 
> The behavior you just encountered sounds like a bug, and it is a rather serious one.
SSTables which should be dead being marked live is very bad for consistency. 
> 
> Do you see any exceptions in your logs or anything? If you can repro, you should file
a JIRA ticket with the apache project...
> 
> =Rob
> 


Mime
View raw message