incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Elias Del Valle <marc...@s1mbi0se.com.br>
Subject Re: too many open files
Date Sat, 09 Aug 2014 04:41:27 GMT
Indeed, that was my mistake, that was exactly what we were doing in the
code.
[]s


2014-08-09 0:56 GMT-03:00 Brian Zhang <yikebocai@gmail.com>:

> For cassandra driver,session is just like database connection pool,it
> maybe contains many tcp connections,if you create a new session every
> time,more and more tcp connections will be created,till surpass the max
> file description limit  of os.
>
> You should create one session,use it repeatedly ,session can manage
> connections automatically,create new connection or close old connection for
> your requests.
>
> 在 2014年8月9日,6:52,Redmumba <redmumba@gmail.com> 写道:
>
> Just to chime in, I also ran into this issue when I was migrating to the
> Datastax client. Instead of reusing the session, I was opening a new
> session each time. For some reason, even though I was still closing the
> session on the client side, I was getting the same error.
>
> Plus, the only way I could recover was by restarting Cassandra. I did not
> really see the connections timeout over a period of a few minutes.
>
> Andrew
> On Aug 8, 2014 3:19 PM, "Andrey Ilinykh" <ailinykh@gmail.com> wrote:
>
>> You may have this problem if your client doesn't reuse the connection but
>> opens new every type. So, run netstat and check the number of established
>> connections. This number should not be big.
>>
>> Thank you,
>>   Andrey
>>
>>
>> On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle <
>> marcelo@s1mbi0se.com.br> wrote:
>>
>>> Hi,
>>>
>>> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having
>>> "too many open files" exceptions when I try to perform a large number of
>>> operations in my 10 node cluster.
>>>
>>> I saw the documentation
>>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
>>> and I have set everything to the recommended settings, but I keep getting
>>> the errors.
>>>
>>> In the documentation it says: "Another, much less likely possibility,
>>> is a file descriptor leak in Cassandra. Run lsof -n | grep java to
>>> check that the number of file descriptors opened by Java is reasonable and
>>> reports the error if the number is greater than a few thousand."
>>>
>>> I guess it's not the case, or else a lot of people would be complaining
>>> about it, but I am not sure what I could do to solve the problem.
>>>
>>> Any hint about how to solve it?
>>>
>>> My client is written in python and uses Cassandra Python Driver. Here
>>> are the exceptions I am having in the client:
>>> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
>>> seconds: Timed out connecting to 200.200.200.144
>>> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
>>> seconds: Timed out connecting to 200.200.200.77
>>>
>>>
>>> And here is the exception I am having in the server:
>>>
>>>  WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
>>> BatchStatement.java (line 223) Batch of prepared statements for
>>> [identification.entity_lookup, identification.entity] is of size 25216,
>>> exceeding specified threshold of 5120 by 20096.
>>> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
>>> ErrorMessage.java (line 222) Unexpected exception during request
>>> java.io.IOException: Connection reset by peer
>>>         at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>>         at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>>>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
>>>         at
>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
>>>         at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
>>>         at
>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>>>         at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>>>         at
>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>         at java.lang.Thread.run(Thread.java:745)
>>>
>>> Here is the amount of open files before and after I restart Cassandra:
>>>
>>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>>> 936580
>>> root@h:/etc/security/limits.d# sudo service cassandra restart
>>> [ ok ] Restarting Cassandra: cassandra.
>>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>>> 80295
>>>
>>>
>>> Best regards,
>>> Marcelo Valle.
>>>
>>
>>
>

Mime
View raw message