incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shiv shivaji <shivaji...@yahoo.com>
Subject Re: Anti-compaction Diskspace issue even when latest patch applied
Date Wed, 24 Feb 2010 18:06:52 GMT
According to the stack trace I get in the log, it makes it look like the patch was for anti-compaction
but I did not look at the source code in detail yet.

java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException: disk full
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
        at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:570)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.UnsupportedOperationException: disk full
        at org.apache.cassandra.db.CompactionManager.doAntiCompaction(CompactionManager.java:344)
        at org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:405)
        at org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:49)
        at org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:130)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        ... 2 more

I tried "nodetool cleanup" before and it did not really stop the disk from filling, is there
a way to force move the data or some other way to solve the issue?


Thanks, Shiv



________________________________
From: Jonathan Ellis <jbellis@gmail.com>
To: cassandra-user@incubator.apache.org
Sent: Wed, February 24, 2010 7:16:32 AM
Subject: Re: Anti-compaction Diskspace issue even when latest patch applied

The patch you refer to was to help *compaction*, not *anticompaction*.

If the space is mostly hints for other machines (is that what you
meant by "due to past problems with others?") you should run nodeprobe
cleanup on it to remove data that doesn't actually belong on that
node.

-Jonathan

On Wed, Feb 24, 2010 at 3:09 AM, shiv shivaji <shivajisus@yahoo.com> wrote:
> For about 6TB of  total data size with a replication factor of 2 (6TB x 2)
> on a five node cluster, I see about 4.6 TB on one machine (due to potential
> past problems with other machines). The machine has a disk of 6TB.
>
> The data folder on this machine has 59,289 files totally 4.6 TB. The files
> are the data, filter and indexes. I see that anti-compaction is running. I
> applied a recent patch which does not do anti-compaction if disk space is
> limited. I still see it happening. I have also called nodetool loadbalance
> on this machine. Seems like it will run out of disk space anyway.
>
> The machine diskspace consumed are: (Each machine has a 6TB hard-drive on
> RAID).
>
> Machine Space Consumed
> M1    4.47 TB
> M2    2.93 TB
> M3    1.83 GB
> M4    56.19 GB
> M5    398.01 GB
>
> How can I force M1 to immediately move its load to M3 and M4 for instance
> (or any other machine). The nodetool move command moves all data, is there a
> way instead to force move 50% of data to M3 and the remaining 50% to M4 and
> resume anti-compaction after the move?
>
> Thanks, Shiv
>
>
>

Mime
View raw message