cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tyler Hobbs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
Date Sat, 10 May 2014 21:57:56 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993824#comment-13993824
] 

Tyler Hobbs commented on CASSANDRA-7042:
----------------------------------------

While trying to reproduce CASSANDRA-6525, I'm seeing similar stacktraces in the logs:

{noformat}
ERROR 13:45:24,236 Exception in thread Thread[CompactionExecutor:7,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /var/lib/cassandra/data/test6981/t12/test6981-t12.t12_blob5_idx-jb-5-Data.db
(No such file or directory)
    at com.google.common.base.Throwables.propagate(Throwables.java:160)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: /var/lib/cassandra/data/test6981/t12/test6981-t12.t12_blob5_idx-jb-5-Data.db
(No such file or directory)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:122)
    at org.apache.cassandra.io.sstable.SSTableReader.preheat(SSTableReader.java:820)
    at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:245)
    at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    ... 8 more
{noformat}

and

{noformat}
ERROR 13:46:42,182 Exception in thread Thread[CompactionExecutor:5,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /var/lib/cassandra/data/test6981/t4/test6981-t4-jb-4-Data.db
(No such file or directory)
    at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
    at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
    at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
    at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
    at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
    at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:262)
    at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:268)
    at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
    at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: /var/lib/cassandra/data/test6981/t4/test6981-t4-jb-4-Data.db
(No such file or directory)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
    at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
    at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
    at org.apache.cassandra.io.compress.CompressedThrottledReader.<init>(CompressedThrottledReader.java:34)
    at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:48)
    ... 17 more
{noformat}

I didn't see any problems when running the 6525 repro script on my laptop, but in a VM with
only 1GB of RAM, I got quite a few of these errors.

> Disk space growth until restart
> -------------------------------
>
>                 Key: CASSANDRA-7042
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>            Reporter: Zach Aller
>         Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot 2014-04-18
at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png, after.log, before.log, tabledump_after_restart.txt,
tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only thing that
seems to fix it is a restart of cassandra this happens about every 3-5 hrs we will grow from
about 350GB to 650GB with no end in site. Once we restart cassandra it usually all clears
itself up and disks return to normal for a while then something triggers its and starts climbing
again. Sometimes when we restart compactions pending skyrocket and if we restart a second
time the compactions pending drop off back to a normal level. One other thing to note is the
space is not free'd until cassandra starts back up and not when shutdown.
> I will get a clean log of before and after restarting next time it happens and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java (line 196)
Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
(No such file or directory)
>         at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
>         at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
>         at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
>         at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
>         at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
>         at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
>         at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
(No such file or directory)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.<init>(Unknown Source)
>         at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
>         at org.apache.cassandra.io.util.ThrottledReader.<init>(ThrottledReader.java:35)
>         at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
>         ... 17 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message