From user-return-25338-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Wed Apr 4 13:57:43 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7DF519D83 for ; Wed, 4 Apr 2012 13:57:43 +0000 (UTC) Received: (qmail 30668 invoked by uid 500); 4 Apr 2012 13:57:41 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 30633 invoked by uid 500); 4 Apr 2012 13:57:41 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 30624 invoked by uid 99); 4 Apr 2012 13:57:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Apr 2012 13:57:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lewis.mcgibbney@gmail.com designates 209.85.161.172 as permitted sender) Received: from [209.85.161.172] (HELO mail-gx0-f172.google.com) (209.85.161.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Apr 2012 13:57:34 +0000 Received: by ggmi1 with SMTP id i1so161150ggm.31 for ; Wed, 04 Apr 2012 06:57:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=VKCLDxWaTU49e43JAKkhkRHvZqlHX5IklVhkAcwUSto=; b=S4XDAYgwkX+9lYVO3y0AMrNVo3E4qNLvInZn7umdCR8420JMiPbwHruDQfsWg1s6fq j7zMVmVpIuKWZi/Sl7eITp+SSMgzAJn7A9sg6jKVEHrLLTIfCbzmBGKNr70iaSHEgAuA WGqTYlcyDIh4nSM3sPGs7ZBASgJ+dtjxy1Yff98HjgvidXPE55Y0XFEEeMfWCSw8run9 fCyc9rDsqSTXsxDGDFZaRJXw/EPAY3IoPzQmYmjwQmxkT/vCpzjdA60Cgk7kU61+Nz9z zht1nmv8AFfpM+FhwOuP2Xllg3cMk33irdnQVt6aOUGkEB73vbsXx04+ONokiKB3IWiO RLMg== MIME-Version: 1.0 Received: by 10.236.109.66 with SMTP id r42mr14678378yhg.39.1333547833992; Wed, 04 Apr 2012 06:57:13 -0700 (PDT) Received: by 10.236.76.162 with HTTP; Wed, 4 Apr 2012 06:57:13 -0700 (PDT) Date: Wed, 4 Apr 2012 14:57:13 +0100 Message-ID: Subject: Attempting to avoid fatal flush due to disk space From: Lewis John Mcgibbney To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf303bfe30efa78f04bcdacc2f X-Virus-Checked: Checked by ClamAV on apache.org --20cf303bfe30efa78f04bcdacc2f Content-Type: text/plain; charset=ISO-8859-1 Hi, When writing some test code (to test our Gora-Cassandra module) for Apache Gora, I'm experiencing a major problem when trying to tearDown, flush and basically clean everything up. The tests consist of running our test suite against an embedded Cassandra instance using the Gora API to do all sorts of tasks which may highkight where there are bugs in the code. Now regardless of whether the tests are failing or not, I get the log and subsequent stack trace as below which quite clearly indicates that there is insufficient disk to flush. However as I don't have the luxury of making changes @ runtime I'm slightly stumped as to how to solve it as all I'm working with is the Cassandra server which is initiated in setUpClass. My yaml is ripped directly from the Cassandra 1.0.2 distribution so there is nothing out of place in there. However maybe there are some settings which I have not configured? Please indicate if I need to provide some more information to paint a clearer picture of the situation. When calling tearDownClass I have if (cassandraThread != null) { cassandraDaemon.stop(); cassandraDaemon.destroy(); cassandraThread.interrupt(); cassandraThread = null; } cleanupDirectoriesFailover(); // this recursively deletes the directories Cassandra works with whilst the server is running. } Thank you very much in advance for any help coming this way. Lewis 12/04/04 13:24:11 INFO gora.GoraTestDriver: tearing down test 12/04/04 13:24:11 INFO migration.Migration: Applying migration 15541bf0-7e51-11e1-0000-242d50cf1fff Drop keyspace: Employee 12/04/04 13:24:11 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Migrations@9190262(6586/82325 serialized/live bytes, 1 ops) 12/04/04 13:24:11 INFO db.Memtable: Writing Memtable-Migrations@9190262(6586/82325 serialized/live bytes, 1 ops) 12/04/04 13:24:11 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Schema@9695615(2796/34950 serialized/live bytes, 2 ops) 12/04/04 13:24:11 INFO db.Memtable: Completed flushing target/test/var/lib/cassandra/data/system/Migrations-h-2764-Data.db (6650 bytes) 12/04/04 13:24:11 INFO db.Memtable: Writing Memtable-Schema@9695615(2796/34950 serialized/live bytes, 2 ops) 12/04/04 13:24:11 INFO db.Memtable: Completed flushing target/test/var/lib/cassandra/data/system/Schema-h-2764-Data.db (2946 bytes) 12/04/04 13:24:12 INFO compaction.CompactionTask: Compacted to [target/test/var/lib/cassandra/data/system/Migrations-h-2765-Data.db,]. 12,979,714 to 12,979,522 (~99% of original) bytes for 1 keys at 12.341213MB/s. Time: 1,003ms. 12/04/04 13:24:12 INFO compaction.CompactionTask: Compacting [SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2764-Data.db'), SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2760-Data.db'), SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2761-Data.db'), SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2763-Data.db')] 12/04/04 13:24:12 INFO store.DataStoreTestBase: tearing down class 12/04/04 13:24:12 INFO service.AbstractCassandraDaemon: Cassandra shutting down... 12/04/04 13:24:12 INFO thrift.CassandraDaemon: Stop listening to thrift clients 12/04/04 13:24:12 INFO compaction.CompactionTask: Compacted to [target/test/var/lib/cassandra/data/system/Schema-h-2765-Data.db,]. 5,735,956 to 5,735,629 (~99% of original) bytes for 1,681 keys at 9.011404MB/s. Time: 607ms. Tests run: 51, Failures: 1, Errors: 50, Skipped: 0, Time elapsed: 913.842 sec <<< FAILURE! 12/04/04 13:25:12 INFO net.MessagingService: Shutting down MessageService... 12/04/04 13:25:12 INFO net.MessagingService: MessagingService shutting down server thread. 12/04/04 13:25:12 INFO net.MessagingService: Waiting for in-progress requests to complete 12/04/04 13:25:12 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Versions@6081747(83/103 serialized/live bytes, 3 ops) 12/04/04 13:25:12 INFO db.Memtable: Writing Memtable-Versions@6081747(83/103 serialized/live bytes, 3 ops) 12/04/04 13:25:12 ERROR service.AbstractCassandraDaemon: Fatal exception in thread Thread[FlushWriter:2,5,main] java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:621) at org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1866) at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:248) at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47) at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more 12/04/04 13:25:12 ERROR service.AbstractCassandraDaemon: Fatal exception in thread Thread[FlushWriter:2,5,main] java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:621) at org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1866) at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:248) at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47) at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more -- *Lewis* --20cf303bfe30efa78f04bcdacc2f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi,

When writing some test code (to test our Gora-Cassandra module) = for Apache Gora, I'm experiencing a major problem when trying to tearDo= wn, flush and basically clean everything up. The tests consist of running o= ur test suite against an embedded Cassandra instance using the Gora API to = do all sorts of tasks which may highkight where there are bugs in the code.=

Now regardless of whether the tests are failing or not, I get the log a= nd subsequent stack trace as below which quite clearly indicates that there= is insufficient disk to flush. However as I don't have the luxury of m= aking changes @ runtime I'm slightly stumped as to how to solve it as a= ll I'm working with is the Cassandra server which is initiated in setUp= Class.

My yaml is ripped directly from the Cassandra 1.0.2 distribution so the= re is nothing out of place in there. However maybe there are some settings = which I have not configured?

Please indicate if I need to provide so= me more information to paint a clearer picture of the situation.

When calling tearDownClass I have

=A0=A0=A0 =A0=A0=A0 if (cassan= draThread !=3D null) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 cassandraDaemon.sto= p();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 cassandraDaemon.destroy();
=A0=A0= =A0 =A0=A0=A0 =A0=A0=A0 cassandraThread.interrupt();
=A0=A0=A0 =A0=A0=A0= =A0=A0=A0 cassandraThread =3D null;
=A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0 cleanupDirectoriesFailover(); = // this recursively deletes the directories Cassandra works with whilst the= server is running.
=A0=A0=A0 }

Thank you very much in advance fo= r any help coming this way.

Lewis

12/04/04 13:24:11 INFO gora.GoraTestDriver: tearing down t= est
12/04/04 13:24:11 INFO migration.Migration: Applying migration 15541= bf0-7e51-11e1-0000-242d50cf1fff Drop keyspace: Employee
12/04/04 13:24:1= 1 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Migrations@9190262= (6586/82325 serialized/live bytes, 1 ops)
12/04/04 13:24:11 INFO db.Memtable: Writing Memtable-Migrations@9190262(658= 6/82325 serialized/live bytes, 1 ops)
12/04/04 13:24:11 INFO db.ColumnFa= milyStore: Enqueuing flush of Memtable-Schema@9695615(2796/34950 serialized= /live bytes, 2 ops)
12/04/04 13:24:11 INFO db.Memtable: Completed flushing target/test/var/lib/= cassandra/data/system/Migrations-h-2764-Data.db (6650 bytes)
12/04/04 13= :24:11 INFO db.Memtable: Writing Memtable-Schema@9695615(2796/34950 seriali= zed/live bytes, 2 ops)
12/04/04 13:24:11 INFO db.Memtable: Completed flushing target/test/var/lib/= cassandra/data/system/Schema-h-2764-Data.db (2946 bytes)
12/04/04 13:24:= 12 INFO compaction.CompactionTask: Compacted to [target/test/var/lib/cassan= dra/data/system/Migrations-h-2765-Data.db,].=A0 12,979,714 to 12,979,522 (~= 99% of original) bytes for 1 keys at 12.341213MB/s.=A0 Time: 1,003ms.
12/04/04 13:24:12 INFO compaction.CompactionTask: Compacting [SSTableReader= (path=3D'target/test/var/lib/cassandra/data/system/Schema-h-2764-Data.d= b'), SSTableReader(path=3D'target/test/var/lib/cassandra/data/syste= m/Schema-h-2760-Data.db'), SSTableReader(path=3D'target/test/var/li= b/cassandra/data/system/Schema-h-2761-Data.db'), SSTableReader(path=3D&= #39;target/test/var/lib/cassandra/data/system/Schema-h-2763-Data.db')]<= br> 12/04/04 13:24:12 INFO store.DataStoreTestBase: tearing down class
12/04= /04 13:24:12 INFO service.AbstractCassandraDaemon: Cassandra shutting down.= ..
12/04/04 13:24:12 INFO thrift.CassandraDaemon: Stop listening to thri= ft clients
12/04/04 13:24:12 INFO compaction.CompactionTask: Compacted to [target/test= /var/lib/cassandra/data/system/Schema-h-2765-Data.db,].=A0 5,735,956 to 5,7= 35,629 (~99% of original) bytes for 1,681 keys at 9.011404MB/s.=A0 Time: 60= 7ms.
Tests run: 51, Failures: 1, Errors: 50, Skipped: 0, Time elapsed: 913.842 s= ec <<< FAILURE!
12/04/04 13:25:12 INFO net.MessagingService: Sh= utting down MessageService...
12/04/04 13:25:12 INFO net.MessagingServic= e: MessagingService shutting down server thread.
12/04/04 13:25:12 INFO net.MessagingService: Waiting for in-progress reques= ts to complete
12/04/04 13:25:12 INFO db.ColumnFamilyStore: Enqueuing fl= ush of Memtable-Versions@6081747(83/103 serialized/live bytes, 3 ops)
12/04/04 13:25:12 INFO db.Memtable: Writing Memtable-Versions@6081747(83/10= 3 serialized/live bytes, 3 ops)
12/04/04 13:25:12 ERROR service.Abstract= CassandraDaemon: Fatal exception in thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk s= pace to flush 133 bytes
=A0=A0=A0 at org.apache.cassandra.utils.WrappedR= unnable.run(WrappedRunnable.java:34)
=A0=A0=A0 at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:908)
=A0=A0=A0 at java.lang.Thread.run(Thread.java:662)
= Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133= bytes
=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnF= amilyStore.java:621)
=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilySt= ore.createFlushWriter(ColumnFamilyStore.java:1866)
=A0=A0=A0 at org.apac= he.cassandra.db.Memtable.writeSortedContents(Memtable.java:248)
=A0=A0=A0 at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)<= br>=A0=A0=A0 at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.jav= a:289)
=A0=A0=A0 at org.apache.cassandra.utils.WrappedRunnable.run(Wrapp= edRunnable.java:30)
=A0=A0=A0 ... 3 more
12/04/04 13:25:12 ERROR service.AbstractCassandraDa= emon: Fatal exception in thread Thread[FlushWriter:2,5,main]
java.lang.R= untimeException: java.lang.RuntimeException: Insufficient disk space to flu= sh 133 bytes
=A0=A0=A0 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable= .java:34)
=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.ru= nTask(ThreadPoolExecutor.java:886)
=A0=A0=A0 at java.util.concurrent.Thr= eadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
=A0=A0=A0 at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.= RuntimeException: Insufficient disk space to flush 133 bytes
=A0=A0=A0 a= t org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.= java:621)
=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(Co= lumnFamilyStore.java:1866)
=A0=A0=A0 at org.apache.cassandra.db.Memtable= .writeSortedContents(Memtable.java:248)
=A0=A0=A0 at org.apache.cassandr= a.db.Memtable.access$400(Memtable.java:47)
=A0=A0=A0 at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:2= 89)
=A0=A0=A0 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:30)
=A0=A0=A0 ... 3 more

--
Lewis

--20cf303bfe30efa78f04bcdacc2f--