Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 247D91142F for ; Thu, 4 Sep 2014 04:15:52 +0000 (UTC) Received: (qmail 21265 invoked by uid 500); 4 Sep 2014 04:15:52 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 21225 invoked by uid 500); 4 Sep 2014 04:15:52 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 21211 invoked by uid 99); 4 Sep 2014 04:15:51 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Sep 2014 04:15:51 +0000 Date: Thu, 4 Sep 2014 04:15:51 +0000 (UTC) From: "Zach Aller (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Issue Comment Deleted] (CASSANDRA-7042) Disk space growth until restart MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach Aller updated CASSANDRA-7042: ---------------------------------- Comment: was deleted (was: Sorry for the late reply, I finally found time to get a build with this patch in it but we are runing DSE and when i try to start apache cassandra i get ERROR 22:28:38,464 Exception encountered during startup java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Unable to find replication strategy class 'org.apache.cassandra.locator.EverywhereStrategy' The only thing i have found on it is (http://qnalist.com/questions/4623138/upgrading-cassandra-from-1-2-11-to-2-0)) > Disk space growth until restart > ------------------------------- > > Key: CASSANDRA-7042 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7042 > Project: Cassandra > Issue Type: Bug > Environment: Ubuntu 12.04 > Sun Java 7 > Cassandra 2.0.6 > Reporter: Zach Aller > Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png, after.log, before.log, system.log.9.gz, tabledump_after_restart.txt, tabledump_before_restart.txt > > > Cassandra will constantly eat disk space not sure whats causing it the only thing that seems to fix it is a restart of cassandra this happens about every 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we restart cassandra it usually all clears itself up and disks return to normal for a while then something triggers its and starts climbing again. Sometimes when we restart compactions pending skyrocket and if we restart a second time the compactions pending drop off back to a normal level. One other thing to note is the space is not free'd until cassandra starts back up and not when shutdown. > I will get a clean log of before and after restarting next time it happens and post it. > Here is a common ERROR in our logs that might be related > {noformat} > ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java (line 196) Exception in thread Thread[CompactionExecutor:46,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db (No such file or directory) > at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53) > at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355) > at org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67) > at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161) > at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173) > at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194) > at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258) > at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126) > at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(Unknown Source) > at org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) > at org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35) > at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49) > ... 17 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)