cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dr. Martin Grabm├╝ller <Martin.Grabmuel...@eleven.de>
Subject RE: Too many open files
Date Fri, 22 Jan 2010 10:36:09 GMT
Sorry, I forgot the following:

I am using a single node, replication factor 1, random partitioner,
and I am doing a multiget_slice with 10 keys.  Yesterday, in the old
setup, I only used get_slice [1].

Maybe Cassandra opens files in parallel for all the keys?

Thanks,
  Martin

[1] The change to version 0.5.0 (pre) was due to that change, because
    of a NPE in the older version for multiget_slice.

> -----Original Message-----
> From: Dr. Martin Grabm├╝ller [mailto:Martin.Grabmueller@eleven.de] 
> Sent: Friday, January 22, 2010 11:27 AM
> To: cassandra-user@incubator.apache.org
> Subject: Too many open files
> 
> Hello all,
> 
> I am using Cassandra for storing mail data, and after
> filling my test installation with data over the night,
> I got "Too many open files" errors.  I checked JIRA for
> known bugs, but found nothing that matched my setup.
> 
> I am using version 0.5.0 from http://people.apache.org/~eevans/
> and the exception looks like this:
> 
> ERROR - Fatal exception in thread Thread[ROW-READ-STAGE:33,5,main]
> java.lang.RuntimeException: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/MessageIndex-844-Data.db 
> (Too many open files)
>         at 
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler
> .java:117)
>         at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDelive
> ryTask.java:38)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx
> ecutor.java:1110)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE
> xecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/MessageIndex-844-Data.db 
> (Too many open files)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
>         at 
> org.apache.cassandra.io.BufferedRandomAccessFile.<init>(Buffer
> edRandomAccessFile.java:142)
>         at 
> org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGrou
> pReader.<init>(SSTableSliceIterator.java:123)
>         at 
> org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SST
> ableSliceIterator.java:58)
>         at 
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColu
> mnIterator(SliceQueryFilter.java:63)
>         at 
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamilyInter
> nal(ColumnFamilyStore.java:1245)
>         at 
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(Colu
> mnFamilyStore.java:1203)
>         at 
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(Colu
> mnFamilyStore.java:1172)
>         at org.apache.cassandra.db.Table.getRow(Table.java:422)
>         at 
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromR
> eadCommand.java:59)
>         at 
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler
> .java:79)
>         ... 4 more
> 
> As can be seen from the trace, the exception happens when trying to
> read from Cassandra.
> 
> I have 789 data files in my data directory (+ the same number
> of index and filter files.
> 
> The obvious call to lsof did not give me any insight (with 2271 being
> my Cassandra instance's pid):
> 
>   (env)cassandra@archive00001:~$ lsof -p 2271|wc -l
>   101
> 
> Maybe the file limit is reached while scanning all data files?
> 
> Nevertheless, when working with a development snapshot yesterday,
> before upgrading, Cassandra was happily handling more than 8000
> data files in its data directory.  That was when I was using snapshot
> apache-cassandra-incubating-2010-01-06_12-47-49 from the snapshot
> download page.
> 
> I am looking for advice on how to debug this.
> 
> Thanks,
>   Martin
> 
Mime
View raw message