cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gurpreet Singh <gurpreet.si...@gmail.com>
Subject Re: disk full error while bootstrapping
Date Thu, 09 Sep 2010 05:50:40 GMT
Thanks Jonathan. I added an extra directory big enough and at least the
bootstrap worked.

However, it brings me to questions that i cant seem to find an answer to.

In my case, i had a 2 node cluster to start with, replication factor of 2,
basically all data residing on both machines. Both nodes had about 150 gigs
of data, and about 30 gigs free space.
Suddenly, one of the machines had a disk failure, so i decided to remove the
node, using nodetool removetoken. Brought up another node to replace the
down node, and tried to bootstrap it. Thats when I ran into the space issue.
Resolved it by adding the extra disk, about 100 gigs.
While bootstrapping I noticed anticompaction happenning for each
columnfamily, and saw the createion of a 92 gig file for the biggest
columnfamily i have, which was then streamed across to the other node. This
file was then deleted.  I followed this up with a nodetool cleanup, as was
mentioned on the wiki to delete any unused references (probably none in my
case, but just wanted to do it!). This triggered an anti-compaction again,
took 2 more hours and created a 92 gig file similar to the last step.
However, nothing happenned after this. This big file just sits there though,
and no other files were deleted. So, now i have 240 gigs of disk used up on
the node where i triggered the cleanup as no files were deleted. When I do a
nodetool ring, it says 93 gigs load, as opposed to 150 gigs previously.

Questions;
1. what is the purpose of this anticompacted file created during cleanup?
2. compaction is also supposed to create a big file after reconciling the
sstable files.. i see that anti-compaction is also creating a big file. How
are these files going to be different?
3. Other than during streaming (bootstrapping), when is anti-compaction
triggered?
4. Now i have 240 gigs of disk used for probably around 92 gigs of useful
data. When will the other files get deleted? Do i need to run compaction
separately to take care of this.

Thanks for your patience,
Gurpreet



On Wed, Sep 8, 2010 at 9:02 AM, Jonathan Ellis <jbellis@gmail.com> wrote:

> if you have a data directory defined with enough room, Cassandra will
> use that one.
>
> On Wed, Sep 8, 2010 at 6:25 AM, Gurpreet Singh <gurpreet.singh@gmail.com>
> wrote:
> > Hi,
> > version: cassandra 0.6.5
> > I am trying to bootstrap a new node from an existing seed node.
> > The new node seems to be stuck with the bootstrapping message, and did
> not
> > show any activity.
> > Only after i checked the logs of the seed node, i realise there has been
> an
> > error:
> > Caused by: java.lang.UnsupportedOperationException: disk full
> > at
> >
> org.apache.cassandra.db.CompactionManager.doAntiCompaction(CompactionManager.java:345)
> > at
> >
> org.apache.cassandra.db.CompactionManager.access$500(CompactionManager.java:49)
> > at
> >
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:143)
> > at
> >
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:140)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > The data size on the seed node is about 140 gigs, and it has abt 20 gig
> free
> > space left.
> > How do i get rid of this error and get the bootstrapping going ?
> > Can it anti-compact to some other place that has disk space available or
> any
> > other ways to get this working?
> > Thanks
> > Gurpreet
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Mime
View raw message