cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dominic Williams (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-2794) Scrub fails on system with blocked compaction with java.io.FileNotFoundException / Too many open files
Date Sat, 18 Jun 2011 14:43:47 GMT
Scrub fails on system with blocked compaction with java.io.FileNotFoundException / Too many
open files
------------------------------------------------------------------------------------------------------

                 Key: CASSANDRA-2794
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2794
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.7.6
         Environment: Ubuntu
            Reporter: Dominic Williams
             Fix For: 0.7.7


A node is suffering from CASSANDRA-2793. Scrub is run to try and fix the problem. Although
ulimit shows unlimited file handles allowed scrub eventually fails with java.io.FileNotFoundException
/ Too many open files:

 INFO [CompactionExecutor:1] 2011-06-17 00:46:26,115 CompactionManager.java (line 511) Scrubbing
SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7018-Data.db')
 INFO [CompactionExecutor:1] 2011-06-17 00:46:26,225 CompactionManager.java (line 652) Scrub
of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7018-Data.db')
complete: 275 rows in new sstable and 0 empty (tombstoned) rows dropped
 INFO [CompactionExecutor:1] 2011-06-17 00:46:26,226 CompactionManager.java (line 511) Scrubbing
SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7580-Data.db')
 INFO [CompactionExecutor:1] 2011-06-17 00:46:28,383 CompactionManager.java (line 652) Scrub
of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7580-Data.db')
complete: 297 rows in new sstable and 0 empty (tombstoned) rows dropped
 INFO [CompactionExecutor:1] 2011-06-17 00:46:28,384 CompactionManager.java (line 511) Scrubbing
SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7574-Data.db')
 INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 CompactionManager.java (line 652) Scrub
of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7574-Data.db')
complete: 347 rows in new sstable and 0 empty (tombstoned) rows dropped
 INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 CompactionManager.java (line 511) Scrubbing
SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7010-Data.db')
ERROR [CompactionExecutor:1] 2011-06-17 00:46:29,374 AbstractCassandraDaemon.java (line 114)
Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/UserMonsters-tmp-f-7823-Data.db
(Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
        at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78)
        at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2243)
        at org.apache.cassandra.db.CompactionManager.maybeCreateWriter(CompactionManager.java:794)
        at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:534)
        at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
        at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

This error eventually "takes over":

ERROR [ReadStage:328] 2011-06-17 00:46:38,634 AbstractCassandraDaemon.java (line 114) Fatal
exception in thread Thread[ReadStage:328,5,main]
java.io.IOError: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3466-Data.db
(Too many open files)
        at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78)
        at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:553)
        at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:95)
        at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1442)
        at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:49)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3466-Data.db
(Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:108)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:93)
        at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:74)
        ... 8 more
ERROR [ReadStage:343] 2011-06-17 00:46:48,624 AbstractCassandraDaemon.java (line 114) Fatal
exception in thread Thread[ReadStage:343,5,main]
java.io.IOError: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3468-Data.db
(Too many open files)
        at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78)
        at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:553)
        at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:95)
        at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1442)
        at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:49)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3468-Data.db
(Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:108)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:93)
        at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:74)
        ... 8 more


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message