cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: cassandra hit a wall: Too many open files (98567!)
Date Mon, 16 Jan 2012 07:35:51 GMT
That sounds like to many sstables. 

Out of interest were you using multi threaded compaction ? Just wondering about this 
https://issues.apache.org/jira/browse/CASSANDRA-3711

Can you set the file handles to unlimited ? 

Can you provide some more info what your see in the data dir incase it is a bug in leveled
compaction. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 14/01/2012, at 8:01 AM, Thorsten von Eicken wrote:

> I'm running a single node cassandra 1.0.6 server which hit a wall yesterday:
> 
> ERROR [CompactionExecutor:2918] 2012-01-12 20:37:06,327
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[CompactionExecutor:2918,1,main] java.io.IOError:
> java.io.FileNotFoundException:
> /mnt/ebs/data/rslog_production/req_word_idx-hc-453661-Data.db (Too many
> open files in system)
> 
> After that it stopped working and just say there with this error
> (undestandable). I did an lsof and saw that it had 98567 open files,
> yikes! An ls in the data directory shows 234011 files. After restarting
> it spent about 5 hours compacting, then quieted down. About 173k files
> left in the data directory. I'm using leveldb (with compression). I
> looked into the json of the two large CFs and gen 0 is empty, most
> sstables are gen 3 & 4. I have a total of about 150GB of data
> (compressed). Almost all the SStables are around 3MB in size. Aren't
> they supposed to get 10x bigger at higher gen's?
> 
> This situation can't be healthy, can it? Suggestions?


Mime
View raw message