incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Tarbox <tar...@cabotresearch.com>
Subject Re: too many open files
Date Mon, 15 Jul 2013 13:16:31 GMT
Odd that this discussion happens now as I'm also getting this error.  I get
a burst of error messages and then the system continues...with no apparent
ill effect.
I can't tell what the system was doing at the time....here is the stack.
 BTW Opscenter says I only have 4 or 5 SSTables in each of my 6 CFs.

ERROR [ReadStage:62384] 2013-07-14 18:04:26,062
AbstractCassandraDaemon.java (line 135) Exception in thread
Thread[ReadStage:62384,5,main]
java.io.IOError: java.io.FileNotFoundException:
/tmp_vol/cassandra/data/dev_a/portfoliodao/dev_a-portfoliodao-hf-166-Data.db
(Too many open files)
        at
org.apache.cassandra.io.util.CompressedSegmentedFile.getSegment(CompressedSegmentedFile.java:69)
        at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:898)
        at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:63)
        at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:61)
        at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
        at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:124)
        at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
        at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
        at org.apache.cassandra.db.Table.getRow(Table.java:378)
        at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
        at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51)
        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException:
/tmp_vol/cassandra/data/dev_a/portfoliodao/dev_a-portfoliodao-hf-166-Data.db
(Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
        at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
        at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:64)
        at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
        at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:41)
        at
org.apache.cassandra.io.util.CompressedSegmentedFile.getSegment(CompressedSegmentedFile.java:63)
        ... 16 more



On Mon, Jul 15, 2013 at 7:23 AM, Michał Michalski <michalm@opera.com> wrote:

> It doesn't tell you anything if file ends it with "ic-###", except
> pointing out the SSTable version it uses ("ic" in this case).
>
> Files related to secondary index contain something like this in the
> filename: <KS>-<CF>.<IDX-NAME>, while in "regular" CFs do not contain
any
> dots except the one just before file extension.
>
> M.
>
> W dniu 15.07.2013 09:38, Paul Ingalls pisze:
>
>  Also, looking through the log, it appears a lot of the files end with
>> ic-#### which I assume is associated with a secondary index I have on the
>> table.  Are secondary indexes really expensive from a file descriptor
>> standpoint?  That particular table uses the default compaction scheme...
>>
>> On Jul 15, 2013, at 12:00 AM, Paul Ingalls <paulingalls@gmail.com> wrote:
>>
>>  I have one table that is using leveled.  It was set to 10MB, I will try
>>> changing it to 256MB.  Is there a good way to merge the existing sstables?
>>>
>>> On Jul 14, 2013, at 5:32 PM, Jonathan Haddad <jon@jonhaddad.com> wrote:
>>>
>>>  Are you using leveled compaction?  If so, what do you have the file
>>>> size set at?  If you're using the defaults, you'll have a ton of really
>>>> small files.  I believe Albert Tobey recommended using 256MB for the table
>>>> sstable_size_in_mb to avoid this problem.
>>>>
>>>>
>>>> On Sun, Jul 14, 2013 at 5:10 PM, Paul Ingalls <paulingalls@gmail.com>
>>>> wrote:
>>>> I'm running into a problem where instances of my cluster are hitting
>>>> over 450K open files.  Is this normal for a 4 node 1.2.6 cluster with
>>>> replication factor of 3 and about 50GB of data on each node?  I can push
>>>> the file descriptor limit up, but I plan on having a much larger load so
>>>> I'm wondering if I should be looking at something else….
>>>>
>>>> Let me know if you need more info…
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Jon Haddad
>>>> http://www.rustyrazorblade.com
>>>> skype: rustyrazorblade
>>>>
>>>
>>>
>>
>>
>

Mime
View raw message