cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thorsten von Eicken <...@rightscale.com>
Subject cassandra hit a wall: Too many open files (98567!)
Date Fri, 13 Jan 2012 19:01:48 GMT
I'm running a single node cassandra 1.0.6 server which hit a wall yesterday:

ERROR [CompactionExecutor:2918] 2012-01-12 20:37:06,327
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[CompactionExecutor:2918,1,main] java.io.IOError:
java.io.FileNotFoundException:
/mnt/ebs/data/rslog_production/req_word_idx-hc-453661-Data.db (Too many
open files in system)

After that it stopped working and just say there with this error
(undestandable). I did an lsof and saw that it had 98567 open files,
yikes! An ls in the data directory shows 234011 files. After restarting
it spent about 5 hours compacting, then quieted down. About 173k files
left in the data directory. I'm using leveldb (with compression). I
looked into the json of the two large CFs and gen 0 is empty, most
sstables are gen 3 & 4. I have a total of about 150GB of data
(compressed). Almost all the SStables are around 3MB in size. Aren't
they supposed to get 10x bigger at higher gen's?

This situation can't be healthy, can it? Suggestions?

Mime
View raw message