Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of sicoe.alexandru@googlemail.com
 designates 209.85.216.51 as permitted sender)
MIME-Version: 1.0
Date: Thu, 1 Dec 2011 11:16:37 +0100
Message-ID: 
 <CACCYQcyG2rmJJzxQ_5YYYp8itxvX1pjey-dtXn1MS35dqjL2Lw@mail.gmail.com>
Subject: Insufficient disk space to flush
From: Alexandru Dan Sicoe <sicoe.alexandru@googlemail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=0016368340b6d0f22a04b305254b

--0016368340b6d0f22a04b305254b
Content-Type: text/plain; charset=ISO-8859-1

Hello everyone,
 4 node Cassandra 0.8.5 cluster with RF =2.
 One node started throwing exceptions in its log:

ERROR 10:02:46,837 Fatal exception in thread Thread[FlushWriter:1317,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk
space to flush 17296 bytes
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush
17296 bytes
        at
org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:714)
        at
org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:2301)
        at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:246)
        at org.apache.cassandra.db.Memtable.access$400(Memtable.java:49)
        at org.apache.cassandra.db.Memtable$3.runMayThrow(Memtable.java:270)
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more

Checked disk and obviously it's 100% full.

How do I recover from this without loosing the data? I've got plenty of
space on the other nodes, so I thought of doing a decommission which I
understand reassigns ranges to the other nodes and replicates data to them.
After that's done I plan on manually deleting the data on the node and then
joining in the same cluster position with auto-bootstrap turned off so that
I won't get back the old data and I can continue getting new data with the
node.

Note, I would like to have 4 nodes in because the other three barely take
the input load alone. These are just long running tests until I get some
better machines.

On strange thing I found is that the data folder on the ndoe that filled up
the disk is 150 GB (as measured with du) while the data folder on all other
3 nodes is 50 GB. At the same time, DataStax OpsCenter shows a size of
around 50GB for all 4 nodes. I though that the node was making a major
compaction at which time it filled up the disk....but even that doesn't
make sense because shouldn't a major compaction just be capable of doubling
the size, not triple-ing it? Doesn anyone know how to explain this behavior?

Thanks,
Alex

--0016368340b6d0f22a04b305254b
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hello everyone,<br>=A04 node Cassandra 0.8.5 cluster with RF =3D2.<br>=A0On=
e node started throwing exceptions in its log: <br><br>ERROR 10:02:46,837 F=
atal exception in thread Thread[FlushWriter:1317,5,main]<br>java.lang.Runti=
meException: java.lang.RuntimeException: Insufficient disk space to flush 1=
7296 bytes<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.utils.WrappedRunnable.run(Wra=
ppedRunnable.java:34)<br>=A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.Thre=
adPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)<br>=A0=A0=A0=A0=
=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE=
xecutor.java:908)<br>
=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:619)<br>Caused by=
: java.lang.RuntimeException: Insufficient disk space to flush 17296 bytes<=
br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilyStore.getFl=
ushPath(ColumnFamilyStore.java:714)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilyStore.createFl=
ushWriter(ColumnFamilyStore.java:2301)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apac=
he.cassandra.db.Memtable.writeSortedContents(Memtable.java:246)<br>=A0=A0=
=A0=A0=A0=A0=A0 at org.apache.cassandra.db.Memtable.access$400(Memtable.jav=
a:49)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.db.Memtable$3.runMayThrow(Mem=
table.java:270)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.utils.Wrap=
pedRunnable.run(WrappedRunnable.java:30)<br>=A0=A0=A0=A0=A0=A0=A0 ... 3 mor=
e<br><br>Checked disk and obviously it&#39;s 100% full.<br>
<br>How do I recover from this without loosing the data? I&#39;ve got plent=
y of space on the other nodes, so I thought of doing a decommission which I=
 understand reassigns ranges to the other nodes and replicates data to them=
. After that&#39;s done I plan on manually deleting the data on the node an=
d then joining in the same cluster position with auto-bootstrap turned off =
so that I won&#39;t get back the old data and I can continue getting new da=
ta with the node.<br>
<br>Note, I would like to have 4 nodes in because the other three barely ta=
ke the input load alone. These are just long running tests until I get some=
 better machines.<br><br clear=3D"all">On strange thing I found is that the=
 data folder on the ndoe that filled up the disk is 150 GB (as measured wit=
h du) while the data folder on all other 3 nodes is 50 GB. At the same time=
, DataStax OpsCenter shows a size of around 50GB for all 4 nodes. I though =
that the node was making a major compaction at which time it filled up the =
disk....but even that doesn&#39;t make sense because shouldn&#39;t a major =
compaction just be capable of doubling the size, not triple-ing it? Doesn a=
nyone know how to explain this behavior?<br>
<br>Thanks,<br>Alex<font size=3D"2"><span style=3D"font-size: 10pt;"><br></=
span></font><br>

--0016368340b6d0f22a04b305254b--