incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: java.io.IOException: No space left on device
Date Wed, 22 Dec 2010 15:20:01 GMT
> And the data could be more evenly balanced, obviously. However the nodes fails to startup
because due of lacking disk space (instead of starting up and denies further writes it appears
to try to process the [6.6G!] commit logs). So, I cannot perform any actions on it no more
like re-balancing the ring or reading old data from it and rotating it somewhere else. So,
what to do now?

So even given deletion of obsolete sstables on start-up, it goes out
of disk just from the commit log replay of only 6 gig? Sounds like
you're very, very full.

Some potential options may be:

* Replace the node completely with a new one with sufficient disk /
another token location (but carefully, by adding the new node first).
* Restarting the nodes with sstables gone and perform a repair to
re-populate it with a minimum amount of overhead (by not writing new
data to it).
* Removing some of the sstables with the aim of allowing commit log
replay to complete, followed by repair.

But those two assume RF > 1 and that you are willing to lose writes
(potentially) according to the consistency level you have used for
writes.

It strikes me that dealing with out-of-disk conditions is probably a
good topic of an in-depth wiki page for what to do in various cases,
depending on usage. The above three suggested options may or may not
be a good idea depending on circumstances. In particular I don't
remember if a repair or bootstrap still, in 0.7, consumes disk-space
on the nodes *from* which data is being sent (anyone?).

In any case: Monitoring disk-space is very very important.

> BTW what precisely does the Owns column mean?

The percentage of the token space owned by the node.

-- 
/ Peter Schuller

Mime
View raw message