Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 40353 invoked from network); 24 Feb 2010 18:07:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 24 Feb 2010 18:07:24 -0000 Received: (qmail 51172 invoked by uid 500); 24 Feb 2010 18:07:24 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 51146 invoked by uid 500); 24 Feb 2010 18:07:24 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 51137 invoked by uid 99); 24 Feb 2010 18:07:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Feb 2010 18:07:24 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [68.142.207.67] (HELO web31804.mail.mud.yahoo.com) (68.142.207.67) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 24 Feb 2010 18:07:14 +0000 Received: (qmail 97262 invoked by uid 60001); 24 Feb 2010 18:06:53 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1267034812; bh=YAWDPFGRH6cnMA6aFUy7OssUguHclvjUOCU0iTnCoaU=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=UCB6qPRZskge7VlehpHfPEn5n88cIOFTencIztNZ7yUUeagE7tvRh/8t/Qwv0Bizg1WWY56TJyo+a/FOt7IhHAkvx4azPNAqVCgUrzk8ap7fCaxnGIepcH5b1J5P60dDWuHu+eoyf2WEsM5jo0rRV7M9RcPfc4iRWdfmWeMU/uQ= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=35wxs8iI3yKvusFSKBX0Ipd1GVd6Fool8nJAvfDIByLN5L2v1iHqGjepJlA/dwqDUSgFrd7ZQYsD4ZIJPHO1EW2Es30IPW3olXYUS+jHdKBUc90g2lfTcV8e1UZW23528yp9CStu1UMvSpb6WqhX8B2NsFXIXDy/nhwk3Y9vHeg=; Message-ID: <906294.96297.qm@web31804.mail.mud.yahoo.com> X-YMail-OSG: NyttNFsVM1lmvsfLba6fQh9iYePQ90a8ihAL89PjJLURIZK J8VMaNVxdgC2lNS2f4BBlJthBYXT_emJc61fm43RW1w3.t2LyOFAb4sRTFeK zQw.azZF.rXQnLQsz08.rbE6480_qdkacDaF6GWEuQYM8NXcOWFWzWA8D4BG ruhZDvRZEccw5r2XpIOWCKlmfUOYbnjDjlqWcwP_YbBDYeZd0gNt.CcdFoEJ fFou6XSRY7q4_88oBBodLCzCHsFgclazpAYOTh4iF88KMIyxJBhyCAsK7qnC blUThXk7P5mWM3FgRPt_2n4LmM_cuyXrlELa98qratALlml11Z0JoEQfux.H 8lLYWYag- Received: from [68.165.4.240] by web31804.mail.mud.yahoo.com via HTTP; Wed, 24 Feb 2010 10:06:52 PST X-Mailer: YahooMailRC/300.3 YahooMailWebService/0.8.100.260964 References: <209311.76109.qm@web31812.mail.mud.yahoo.com> Date: Wed, 24 Feb 2010 10:06:52 -0800 (PST) From: shiv shivaji Subject: Re: Anti-compaction Diskspace issue even when latest patch applied To: cassandra-user@incubator.apache.org In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-686195376-1267034812=:96297" X-Virus-Checked: Checked by ClamAV on apache.org --0-686195376-1267034812=:96297 Content-Type: text/plain; charset=us-ascii According to the stack trace I get in the log, it makes it look like the patch was for anti-compaction but I did not look at the source code in detail yet. java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException: disk full at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:570) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.UnsupportedOperationException: disk full at org.apache.cassandra.db.CompactionManager.doAntiCompaction(CompactionManager.java:344) at org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:405) at org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:49) at org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:130) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ... 2 more I tried "nodetool cleanup" before and it did not really stop the disk from filling, is there a way to force move the data or some other way to solve the issue? Thanks, Shiv ________________________________ From: Jonathan Ellis To: cassandra-user@incubator.apache.org Sent: Wed, February 24, 2010 7:16:32 AM Subject: Re: Anti-compaction Diskspace issue even when latest patch applied The patch you refer to was to help *compaction*, not *anticompaction*. If the space is mostly hints for other machines (is that what you meant by "due to past problems with others?") you should run nodeprobe cleanup on it to remove data that doesn't actually belong on that node. -Jonathan On Wed, Feb 24, 2010 at 3:09 AM, shiv shivaji wrote: > For about 6TB of total data size with a replication factor of 2 (6TB x 2) > on a five node cluster, I see about 4.6 TB on one machine (due to potential > past problems with other machines). The machine has a disk of 6TB. > > The data folder on this machine has 59,289 files totally 4.6 TB. The files > are the data, filter and indexes. I see that anti-compaction is running. I > applied a recent patch which does not do anti-compaction if disk space is > limited. I still see it happening. I have also called nodetool loadbalance > on this machine. Seems like it will run out of disk space anyway. > > The machine diskspace consumed are: (Each machine has a 6TB hard-drive on > RAID). > > Machine Space Consumed > M1 4.47 TB > M2 2.93 TB > M3 1.83 GB > M4 56.19 GB > M5 398.01 GB > > How can I force M1 to immediately move its load to M3 and M4 for instance > (or any other machine). The nodetool move command moves all data, is there a > way instead to force move 50% of data to M3 and the remaining 50% to M4 and > resume anti-compaction after the move? > > Thanks, Shiv > > > --0-686195376-1267034812=:96297 Content-Type: text/html; charset=us-ascii
According to the stack trace I get in the log, it makes it look like the patch was for anti-compaction but I did not look at the source code in detail yet.

java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException: disk full
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
        at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:570)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.UnsupportedOperationException: disk full
        at org.apache.cassandra.db.CompactionManager.doAntiCompaction(CompactionManager.java:344)
        at org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:405)
        at org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:49)
        at org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:130)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        ... 2 more

I tried "nodetool cleanup" before and it did not really stop the disk from filling, is there a way to force move the data or some other way to solve the issue?

Thanks, Shiv

From: Jonathan Ellis <jbellis@gmail.com>
To: cassandra-user@incubator.apache.org
Sent: Wed, February 24, 2010 7:16:32 AM
Subject: Re: Anti-compaction Diskspace issue even when latest patch applied

The patch you refer to was to help *compaction*, not *anticompaction*.

If the space is mostly hints for other machines (is that what you
meant by "due to past problems with others?") you should run nodeprobe
cleanup on it to remove data that doesn't actually belong on that
node.

-Jonathan

On Wed, Feb 24, 2010 at 3:09 AM, shiv shivaji <shivajisus@yahoo.com> wrote:
> For about 6TB of  total data size with a replication factor of 2 (6TB x 2)
> on a five node cluster, I see about 4.6 TB on one machine (due to potential
> past problems with other machines). The machine has a disk of 6TB.
>
> The data folder on this machine has 59,289 files totally 4.6 TB. The files
> are the data, filter and indexes. I see that anti-compaction is running. I
> applied a recent patch which does not do anti-compaction if disk space is
> limited. I still see it happening. I have also called nodetool loadbalance
> on this machine. Seems like it will run out of disk space anyway.
>
> The machine diskspace consumed are: (Each machine has a 6TB hard-drive on
> RAID).
>
> Machine Space Consumed
> M1    4.47 TB
> M2    2.93 TB
> M3    1.83 GB
> M4    56.19 GB
> M5    398.01 GB
>
> How can I force M1 to immediately move its load to M3 and M4 for instance
> (or any other machine). The nodetool move command moves all data, is there a
> way instead to force move 50% of data to M3 and the remaining 50% to M4 and
> resume anti-compaction after the move?
>
> Thanks, Shiv
>
>
>
--0-686195376-1267034812=:96297--