Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 27 Mar 2015 13:41:53 +0000 (UTC)
From: "Erik Forsberg (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12785481.1427281190000.49855.1427463713133@Atlassian.JIRA>
In-Reply-To: <JIRA.12785481.1427281190000@Atlassian.JIRA>
References: <JIRA.12785481.1427281190000@Atlassian.JIRA>
 <JIRA.12785481.1427281190417@arcas>
Subject: [jira] [Commented] (CASSANDRA-9036) "disk full" when running
 cleanup (on a far from full disk)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383840#comment-14383840 ] 

Erik Forsberg commented on CASSANDRA-9036:
------------------------------------------

Applied the 2.0 version and put in production on one of my nodes. Cleanup of my rather-large-file now started without exception. 

So I'm now officially happy! :-)

> "disk full" when running cleanup (on a far from full disk)
> ----------------------------------------------------------
>
>                 Key: CASSANDRA-9036
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9036
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Erik Forsberg
>            Assignee: Robert Stupp
>         Attachments: 9036-2.0.txt, 9036-2.1.txt, 9036-3.0.txt
>
>
> I'm trying to run cleanup, but get this:
> {noformat}
>  INFO [CompactionExecutor:18] 2015-03-25 10:29:16,355 CompactionManager.java (line 564) Cleaning up SSTableReader(path='/cassandra/production/Data_daily/production-Data_daily-jb-4345750-Data.db')
> ERROR [CompactionExecutor:18] 2015-03-25 10:29:16,664 CassandraDaemon.java (line 199) Exception in thread Thread[CompactionExecutor:18,1,main]
> java.io.IOException: disk full
>         at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:567)
>         at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63)
>         at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281)
>         at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Now that's odd, since:
> * Disk has some 680G left
> * The sstable it's trying to cleanup is far less than 680G:
> {noformat}
> # ls -lh *4345750*
> -rw-r--r-- 1 cassandra cassandra  64M Mar 21 04:42 production-Data_daily-jb-4345750-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 219G Mar 21 04:42 production-Data_daily-jb-4345750-Data.db
> -rw-r--r-- 1 cassandra cassandra 503M Mar 21 04:42 production-Data_daily-jb-4345750-Filter.db
> -rw-r--r-- 1 cassandra cassandra  42G Mar 21 04:42 production-Data_daily-jb-4345750-Index.db
> -rw-r--r-- 1 cassandra cassandra 5.9K Mar 21 04:42 production-Data_daily-jb-4345750-Statistics.db
> -rw-r--r-- 1 cassandra cassandra  81M Mar 21 04:42 production-Data_daily-jb-4345750-Summary.db
> -rw-r--r-- 1 cassandra cassandra   79 Mar 21 04:42 production-Data_daily-jb-4345750-TOC.txt
> {noformat}
> Sure, it's large, but it's not 680G. 
> No other compactions are running on that server. I'm getting this on 12 / 56 servers right now. 
> Could it be some bug in the calculation of the expected size of the new sstable, perhaps? 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)