incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <>
Subject Re: repair takes two days, and ends up stuck: stream at 1096% (yes, really)
Date Mon, 15 Nov 2010 22:01:13 GMT
On Mon, Nov 15, 2010 at 3:05 PM, Reverend Chip <> wrote:
> On 11/15/2010 12:09 PM, Jonathan Ellis wrote:
>> On Mon, Nov 15, 2010 at 1:03 PM, Reverend Chip <> wrote:
>>> I find X.21's data disk is full.  "nodetool ring" says that X.21 has a
>>> load of only 326.2 GB, but the 1T partition is full.
>> Load only tracks live data -- is the rest tmp files?
> No, there are a lot of non-tmps that were not included in the load
> figure.  Having stopped the server and deleted tmp files, the data are
> still using way more space than "ring" claimed -- and too much for
> "cleanup" to work, as well:
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/flashcache
>                      932G  723G  162G  82% /var/lib/cassandra/data

I take it you restarted it?  Does it show the correct load now?

> Given that the previous situation included incomplete replication, I
> can't just kill the node and let it repopulate.  So I can either magick
> up more disk space or reload the whole cluster.  :-(

You can move the data files do a node with more space, compact there,
and move them back, but if you don't care about the data rebuilding is
more straightforward.

> Is there anything about the node's data directory that you need to see?
> Or is it reload time?

I don't think so, thanks.

Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support

View raw message