incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From elsif <elsif.t...@gmail.com>
Subject Leaked FDs
Date Fri, 13 Nov 2009 00:28:25 GMT
Hello.  I am testing out Cassandra and ran out of file descriptors while
attempting to scan through the key space.  I have ~180 million unique
keys, 10 nodes, and a replication factor of 3.

Cassandra reported the following:

WARN [GMFD:1] 2009-11-12 16:07:24,961 MessagingService.java (line 393)
Exception was generated at : 11/12/2009 16:07:24 on thread GMFD:1
Too many open files
java.net.SocketException: Too many open files
        at sun.nio.ch.Net.socket0(Native Method)
        at sun.nio.ch.Net.socket(Unknown Source)
        at sun.nio.ch.DatagramChannelImpl.<init>(Unknown Source)
        at sun.nio.ch.SelectorProviderImpl.openDatagramChannel(Unknown
Source)
        at java.nio.channels.DatagramChannel.open(Unknown Source)
        at
org.apache.cassandra.net.UdpConnection.init(UdpConnection.java:49)
        at
org.apache.cassandra.net.MessagingService.sendUdpOneWay(MessagingService.java:388)
        at
org.apache.cassandra.gms.GossipDigestSynVerbHandler.doVerb(Gossiper.java:889)
        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
        at java.lang.Thread.run(Unknown Source)

Lsof reports that the java process has 65486 files open (I have ulimit
-n 65535 set in cassandra.in.sh).  Many of the lsof entries include a
trailing '(deleted)' comment after the file path.

This appears to be similar to CASSANDRA-283.  Anyone have a work around
for this?  Would a forced GC take care of the ones marked deleted?

Here is my sample code to count the number of keys:

public class CClient
{
    public static void main(String[] args)
    throws TException, InvalidRequestException, UnavailableException,
UnsupportedEncodingException, NotFoundException
    {
        TTransport tr = new TSocket("localhost", 9160);
        TProtocol proto = new TBinaryProtocol(tr);
        Cassandra.Client client = new Cassandra.Client(proto);
        tr.open();

        int     count = 0;
        int     block = 1000;
        String  key   = " ";

        while (true)
        {
            List<String> list = client.get_key_range("Keyspace1",
                "Standard1", key, "~", block, ConsistencyLevel.ONE);
            int size = list.size();
            if (size == 0)
                break;
            count += size;
            key = list.get(size - 1) + '~';
            System.out.println("Count: " + Integer.toString(count));
       }
        tr.close();
    }
}


Mime
View raw message