incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yatong Zhang <bluefl...@gmail.com>
Subject Cassandra 2.0.7 always failes due to 'too may open files' error
Date Mon, 05 May 2014 00:58:20 GMT
Hi there,

After I changed compaction strategy to 'leveled', one of my nodes keeps
reporting 'too many open files. But I have done some configuration
following: http://www.datastax.com/docs/1.1/install/recommended_settingsand
http://www.datastax.com/docs/1.1/troubleshooting/index#toomany

I am using 2.0.7 and here is the system info:

[root@storage5 ~]# ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 126550
> max locked memory       (kbytes, -l) unlimited
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 999999
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 10240
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 126550
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> [root@storage5 ~]# sysctl -a | grep fs.file-max
> fs.file-max = 9999999
> [root@storage5 ~]# lsof -n | grep java | wc -l
> 5103
> [root@storage5 ~]# lsof | wc -l
> 6567
>



Here is the error:


> ERROR [CompactionExecutor:1] 2014-05-05 08:33:56,077 CassandraDaemon.java
> (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
> FSWriteError in /data6/cass/mydb/images/mydb-images-tmp-jb-109136-Filter.db
>         at
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
>         at
> org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
>         at
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
>         at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209)
>         at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>         at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>         at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.FileNotFoundException:
> /data6/cass/mydb/images/mydb-images-tmp-jb-109136-Filter.db (Too many open
> files)
>         at java.io.FileOutputStream.open(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:110)
>         at
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:466)
>         ... 13 more
> ERROR [CompactionExecutor:1] 2014-05-05 08:33:56,105 StorageService.java
> (line 367) Stopping gossiper
>

Mime
View raw message