cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josep M. Blanquer (JIRA)" <>
Subject [jira] Created: (CASSANDRA-1858) File descriptors to sstables not closed (even for sstables that have been deleted due to compaction)
Date Tue, 14 Dec 2010 00:58:00 GMT
File descriptors to sstables not closed (even for sstables that have been deleted due to compaction)

                 Key: CASSANDRA-1858
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.7.0 rc 1
         Environment: Ubuntu 8.04 with Java 1.6.0_22-b04
            Reporter: Josep M. Blanquer

The Cassandra process doesn't let go of filedescriptors to sstables.
It was initially spotted due to the fact that the disk utilization was really high, while
the data dirs of sstables held much less (nicely compacted) data.
Performing an "lsof" lists the Cassandra process having basically all open FD's pointing to
lots and lots of deleted files. In fact, I saw the "-1-Data.db" in the list...which tells
me that not even the first sstable was let go.

Restarting the cassandra process obviously gets rid of the references. I've also manually
triggered a GC to try to make sure compaction is no avail. Holding FD's for
a high-throughput write servers is a big problem since it accumulates a huge amount of disk
space due to it's very easy to run out of space.

There's nothing special in this setup, so I think it'd be easily reproducible. I get this
in a:
* 4-node cluster, 1 Keyspace, 1 CF, RF=2
* the cluster has basically been stood up, and blasted with inserts (now it has 200GB, 50
each)...but it doesn't get reads (I'm performing just loading data tests).
* While the loading is happening, one can observe that the process keeps compacting and creating
new sstables...but never lets the FD's go.
* even if I trigger compactions and cleanups...the disk util only continues to go up...i.e.,
all FD's are still open, plus the compated sstables are added.
* at a particular node I see about 11 fd's for current sstables, and 1250 fd's pointing to
deleted sstables.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message