cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Williams <je...@wherethebitsroam.com>
Subject Re: Compaction issues, 2.0.12
Date Tue, 07 Jul 2015 10:07:15 GMT
Thanks Rob, Jeff. I have updated the Jira issue with my information.

On 6 July 2015 at 23:46, Jeff Ferland <jbf@tubularlabs.com> wrote:

> I’ve seen the same thing:
> https://issues.apache.org/jira/browse/CASSANDRA-9577
>
> I’ve had cases where a restart clears the old tables, and I’ve had cases
> where a restart considers the old tables to be live.
>
> On Jul 6, 2015, at 1:51 PM, Robert Coli <rcoli@eventbrite.com> wrote:
>
> On Mon, Jul 6, 2015 at 1:46 PM, Jeff Williams <jeffw@wherethebitsroam.com>
> wrote:
>
>> 1) Cassandra version is 2.0.12.
>>
>
>
>> 2) Interesting. Looking at JMX org.apache.cassandra.db -> ColumnFamilies
>> -> trackcontent -> track_content -> Attributes, I get:
>>
>> LiveDiskSpaceUsed: 17788740448, i.e. ~17GB
>> LiveSSTableCount: 3
>> TotalDiskSpaceUsed: 55714084629, i.e. ~55GB
>>
>> So it obviously knows about the extra disk space even though the "live"
>> space looks correct. I couldn't find anything to identify the actual files
>> though.
>>
>
> That's what I would expect.
>
>
>> 3) So that was even more interesting. After restarting the cassandra
>> daemon, the sstables were not deleted and now the same JMX attributes are:
>>
>> LiveDiskSpaceUsed: 55681040579, i.e. ~55GB
>> LiveSSTableCount: 8
>> TotalDiskSpaceUsed: 55681040579, i.e. ~55GB
>>
>> So some of my non-live tables are back live again, and obviously some of
>> the big ones!!
>>
>
> This is permanently fatal to consistency; sorry if I was not clear enough
> that if they were not live, there was some risk of Cassandra considering
> them live again upon restart.
>
> If I were you, I would either stop the node and remove the files you know
> shouldn't be live or do a major compaction ASAP.
>
> The behavior you just encountered sounds like a bug, and it is a rather
> serious one. SSTables which should be dead being marked live is very bad
> for consistency.
>
> Do you see any exceptions in your logs or anything? If you can repro, you
> should file a JIRA ticket with the apache project...
>
> =Rob
>
>
>

Mime
View raw message