cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anthony Fisk (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-8964) SSTable count rises during compactions and max open files exceeded
Date Thu, 12 Mar 2015 22:17:38 GMT
Anthony Fisk created CASSANDRA-8964:
---------------------------------------

             Summary: SSTable count rises during compactions and max open files exceeded
                 Key: CASSANDRA-8964
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8964
             Project: Cassandra
          Issue Type: Bug
         Environment: Apache Cassandra 2.1.2
Centos 6
AWS EC2
i2.2xlarge
            Reporter: Anthony Fisk
            Priority: Critical
         Attachments: lsof_with_tmp.txt, lsof_without_tmp.txt, nodetool_cfstats.zip

LCS compaction was not able to keep up with the prolonged insert load on one of our tables
called log, resulting in 2,185 SSTables for that table and 1,779 pending compactions all together
during a test we were running.

We stopped our load, unthrottled compaction throughput, increased the concurrent compactors
from 2 to 8, and let it compact the SSTables.

All was going well until the number of SSTables count for our log table got down to around
97, then began rising again until it had reached 758 SSTables 1.5 hours later... (we've been
recording the cfstats output every half hour, [attached|nodetool_cfstats.zip])

Eventually we exceeded the number of open files:
{code}
ERROR [MemtableFlushWriter:286] 2015-03-12 13:44:36,748 CassandraDaemon.java:153 - Exception
in thread Thread[MemtableFlushWriter:286,5,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-6618-Index.db
(Too many open files)
        at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:75)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:104) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:552)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:134)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:390)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:329)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:313) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.jar:na]
        at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1037)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_51]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_51]
        at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_51]
Caused by: java.io.FileNotFoundException: /mnt/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-6618-Index.db
(Too many open files)
        at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_51]
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241) ~[na:1.7.0_51]
        at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:71)
~[apache-cassandra-2.1.2.jar:2.1.2]
        ... 14 common frames omitted
ERROR [MemtableFlushWriter:286] 2015-03-12 13:44:36,750 JVMStabilityInspector.java:94 - JVM
state determined to be unstable.  Exiting forcefully due to:
java.io.FileNotFoundException: /mnt/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-6618-Index.db
(Too many open files)
        at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_51]
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241) ~[na:1.7.0_51]
        at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:71)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:104) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:552)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:134)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:390)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:329)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:313) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.2.jar:2.1.2]
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.jar:na]
        at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1037)
~[apache-cassandra-2.1.2.jar:2.1.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_51]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_51]
        at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_51]
{code}

I restarted Cassandra and the number of SSTables reduced to 81 before increasing again...
Looking at lsof output, it seemed that the file handles for the log table were also continually
increasing. It seemed that roughly half of them [were for "tmp" files|lsof_with_tmp.txt] and
the other half [were for the directory of the table|lsof_without_tmp.txt]. Also of note, it
seemed we only had one compactor running on the log table, rather than the 8 that we had seen
before after changing the configuration and restarting.

Cassandra eventually crashed again with the same error when it reached 6,442 open files (seen
in lsof). Without Cassandra running our system has 1,126 open files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message