cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zach Aller (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-7042) Disk space growth until restart
Date Thu, 24 Apr 2014 18:37:20 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zach Aller updated CASSANDRA-7042:
----------------------------------

    Attachment: tabledump_before_restart.txt
                tabledump_after_restart.txt

This is a dump of the directory for the cf that has the issues as you can see file count increases.
It seems like files are not being deleted when they should be. The before file is taken when
cassandra was in a stopped state the after is when cassandra had been started up again.

> Disk space growth until restart
> -------------------------------
>
>                 Key: CASSANDRA-7042
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>            Reporter: Zach Aller
>         Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot 2014-04-18
at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png, after.log, before.log, tabledump_after_restart.txt,
tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only thing that
seems to fix it is a restart of cassandra this happens about every 3-5 hrs we will grow from
about 350GB to 650GB with no end in site. Once we restart cassandra it usually all clears
itself up and disks return to normal for a while then something triggers its and starts climbing
again. Sometimes when we restart compactions pending skyrocket and if we restart a second
time the compactions pending drop off back to a normal level. One other thing to note is the
space is not free'd until cassandra starts back up and not when shutdown.
> I will get a clean log of before and after restarting next time it happens and post it.
> Here is a common ERROR in our logs that might be related
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java (line 196)
Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
(No such file or directory)
>         at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
>         at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
>         at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
>         at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
>         at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
>         at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
>         at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
(No such file or directory)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.<init>(Unknown Source)
>         at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
>         at org.apache.cassandra.io.util.ThrottledReader.<init>(ThrottledReader.java:35)
>         at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
>         ... 17 more



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message