Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Message-ID: <6520061.104871292288280998.JavaMail.jira@thor>
Date: Mon, 13 Dec 2010 19:58:00 -0500 (EST)
From: "Josep M. Blanquer (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Subject: [jira] Created: (CASSANDRA-1858) File descriptors to sstables not
 closed (even for sstables that have been deleted due to compaction)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

File descriptors to sstables not closed (even for sstables that have been deleted due to compaction)
----------------------------------------------------------------------------------------------------

                 Key: CASSANDRA-1858
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1858
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.7.0 rc 1
         Environment: Ubuntu 8.04 with Java 1.6.0_22-b04
            Reporter: Josep M. Blanquer


The Cassandra process doesn't let go of filedescriptors to sstables.
It was initially spotted due to the fact that the disk utilization was really high, while the data dirs of sstables held much less (nicely compacted) data.
Performing an "lsof" lists the Cassandra process having basically all open FD's pointing to lots and lots of deleted files. In fact, I saw the "-1-Data.db" in the list...which tells me that not even the first sstable was let go.

Restarting the cassandra process obviously gets rid of the references. I've also manually triggered a GC to try to make sure compaction is completed...to no avail. Holding FD's for a high-throughput write servers is a big problem since it accumulates a huge amount of disk space due to compactions..etc...so it's very easy to run out of space.


There's nothing special in this setup, so I think it'd be easily reproducible. I get this in a:
* 4-node cluster, 1 Keyspace, 1 CF, RF=2
* the cluster has basically been stood up, and blasted with inserts (now it has 200GB, 50 each)...but it doesn't get reads (I'm performing just loading data tests).
* While the loading is happening, one can observe that the process keeps compacting and creating new sstables...but never lets the FD's go.
* even if I trigger compactions and cleanups...the disk util only continues to go up...i.e., all FD's are still open, plus the compated sstables are added.
* at a particular node I see about 11 fd's for current sstables, and 1250 fd's pointing to deleted sstables.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.