cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Williams (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9577) Cassandra not performing GC on stale SStables after compaction
Date Tue, 07 Jul 2015 08:20:05 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616361#comment-14616361
] 

Jeff Williams commented on CASSANDRA-9577:
------------------------------------------

I am the user mentioned by Rob above. I have run 3 major compactions now and after each compaction
it creates a new sstable, but leaves the old. About 5 hours after the compaction, JMX said
I had 3 live sstables, though there were 14 in the data directory. After restarting cassandra
it reported 8 live sstables, so some came back from the dead.

Right now, JMX is reporting 10 sstables and there are 10 sstables in the data directory, but
with 6 of them from before the last major compaction. 57372 was the result of the last compaction.

$ ls -l /var/lib/cassandra/data/trackcontent/track_content/*Data.db
-rw-r--r-- 2 cassandra cassandra 16642164891 Jun 29 06:55 /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30852-Data.db
-rw-r--r-- 2 cassandra cassandra 17216513377 Jun 30 08:36 /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-30866-Data.db
-rw-r--r-- 2 cassandra cassandra   813683923 Jun 30 12:03 /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-37136-Data.db
-rw-r--r-- 2 cassandra cassandra   855070477 Jun 30 13:15 /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-43371-Data.db
-rw-r--r-- 2 cassandra cassandra   209921645 Jun 30 13:28 /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-44676-Data.db
-rw-r--r-- 2 cassandra cassandra   213532360 Jul  2 03:16 /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-51845-Data.db
-rw-r--r-- 2 cassandra cassandra 16916868412 Jul  6 13:34 /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57372-Data.db
-rw-r--r-- 2 cassandra cassandra   256127620 Jul  6 20:27 /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57377-Data.db
-rw-r--r-- 2 cassandra cassandra     8740917 Jul  7 02:10 /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57378-Data.db
-rw-r--r-- 2 cassandra cassandra     1371689 Jul  7 03:23 /var/lib/cassandra/data/trackcontent/track_content/trackcontent-track_content-jb-57379-Data.db

How is this bad for consistency? I have been assuming (perhaps incorrectly) that reads will
hit the new sstables first, so they should get the correct data without hitting the old sstables.

I now have 3 copies of the dataset stored for this table on this node and am getting close
to filling the disk (70%). I am happy to just remove the node, but I am wondering if you want
me to try something first?

I do not see any errors in the logs, except for a exception due to "disk full" when I tried
a "nodetool cleanup" after adding a new node to the cluster. I am assuming that this is related.

> Cassandra not performing GC on stale SStables after compaction
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-9577
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9577
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: 2.0.12.200 / DSE 4.6.1.
>            Reporter: Jeff Ferland
>            Assignee: Marcus Eriksson
>
> 		Space used (live), bytes:   878681716067
> 		Space used (total), bytes: 2227857083852
> jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ sudo lsof *-Data.db 
> COMMAND  PID      USER   FD   TYPE DEVICE     SIZE/OFF  NODE NAME
> java    4473 cassandra  446r   REG   0,26  17582559172 39241 trends-trends-jb-144864-Data.db
> java    4473 cassandra  448r   REG   0,26     62040962 37431 trends-trends-jb-144731-Data.db
> java    4473 cassandra  449r   REG   0,26 829935047545 21150 trends-trends-jb-143581-Data.db
> java    4473 cassandra  452r   REG   0,26      8980406 39503 trends-trends-jb-144882-Data.db
> java    4473 cassandra  454r   REG   0,26      8980406 39503 trends-trends-jb-144882-Data.db
> java    4473 cassandra  462r   REG   0,26      9487703 39542 trends-trends-jb-144883-Data.db
> java    4473 cassandra  463r   REG   0,26     36158226 39629 trends-trends-jb-144889-Data.db
> java    4473 cassandra  468r   REG   0,26    105693505 39447 trends-trends-jb-144881-Data.db
> java    4473 cassandra  530r   REG   0,26  17582559172 39241 trends-trends-jb-144864-Data.db
> java    4473 cassandra  535r   REG   0,26    105693505 39447 trends-trends-jb-144881-Data.db
> java    4473 cassandra  542r   REG   0,26      9487703 39542 trends-trends-jb-144883-Data.db
> java    4473 cassandra  553u   REG   0,26   6431729821 39556 trends-trends-tmp-jb-144884-Data.db
> jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ ls *-Data.db
> trends-trends-jb-142631-Data.db  trends-trends-jb-143562-Data.db  trends-trends-jb-143581-Data.db
 trends-trends-jb-144731-Data.db  trends-trends-jb-144883-Data.db
> trends-trends-jb-142633-Data.db  trends-trends-jb-143563-Data.db  trends-trends-jb-144530-Data.db
 trends-trends-jb-144864-Data.db  trends-trends-jb-144889-Data.db
> trends-trends-jb-143026-Data.db  trends-trends-jb-143564-Data.db  trends-trends-jb-144551-Data.db
 trends-trends-jb-144881-Data.db  trends-trends-tmp-jb-144884-Data.db
> trends-trends-jb-143533-Data.db  trends-trends-jb-143578-Data.db  trends-trends-jb-144552-Data.db
 trends-trends-jb-144882-Data.db
> jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ cd -
> /mnt/cassandra/data/trends/trends
> jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ sudo lsof * 
> jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ ls *-Data.db
> trends-trends-jb-124502-Data.db  trends-trends-jb-141113-Data.db  trends-trends-jb-141377-Data.db
 trends-trends-jb-141846-Data.db  trends-trends-jb-144890-Data.db
> trends-trends-jb-125457-Data.db  trends-trends-jb-141123-Data.db  trends-trends-jb-141391-Data.db
 trends-trends-jb-141871-Data.db  trends-trends-jb-41121-Data.db
> trends-trends-jb-130016-Data.db  trends-trends-jb-141137-Data.db  trends-trends-jb-141538-Data.db
 trends-trends-jb-141883-Data.db  trends-trends.trends_date_idx-jb-2100-Data.db
> trends-trends-jb-139563-Data.db  trends-trends-jb-141358-Data.db  trends-trends-jb-141806-Data.db
 trends-trends-jb-142033-Data.db
> trends-trends-jb-141102-Data.db  trends-trends-jb-141363-Data.db  trends-trends-jb-141829-Data.db
 trends-trends-jb-144553-Data.db
> Compaction started  INFO [CompactionExecutor:6661] 2015-06-05 14:02:36,515 CompactionTask.java
(line 120) Compacting [SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-124502-Data.db'),
SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141358-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141883-Data.db'),
SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141846-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141871-Data.db'),
SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141391-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-139563-Data.db'),
SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-125457-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141806-Data.db'),
SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141102-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141123-Data.db'),
SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-41121-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141137-Data.db'),
SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141113-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-142033-Data.db'),
SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141829-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141377-Data.db'),
SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141538-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141363-Data.db'),
SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-130016-Data.db')]
> Compaction ended:  INFO [CompactionExecutor:6661] 2015-06-09 06:54:27,527 CompactionTask.java
(line 296) Compacted 20 sstables to [/ebs/cassandra/data/trends/trends/trends-trends-jb-143581,].
 1,260,539,306,144 bytes to 829,935,047,545 (~65% of original) in 319,910,989ms = 2.474087MB/s.
 1,913,263,636 total partitions merged to 873,858,309.  Partition merge counts were {1:391213541,
2:241603436, 3:102139024, 4:52438965, 5:47749892, 6:13962210, 7:9923980, 8:8363815, 9:3626260,
10:1413014, 11:615314, 12:336503, 13:416749, 14:55606, }
> date: Wed Jun 10 23:43:31 UTC 2015



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message