incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yatong Zhang <bluefl...@gmail.com>
Subject Re: Cassandra 2.0.7 always failes due to 'too may open files' error
Date Fri, 16 May 2014 13:41:40 GMT
Yes the global limits are OK. I added cassandra to '/etc/rc.local' to make
it auto-startup, but seems the modification of limits didn't take effect. I
observed this as Bryan suggested, so I added

ulimit -SHn 999999
>

to '/etc/rc.local' and before cassandra start command, and it worked.


On Thu, May 8, 2014 at 3:34 AM, Nikolay Mihaylov <nmmm@nmmm.nu> wrote:

> sorry, probably somebody mentioned it, but did you checked global limit?
>
> cat /proc/sys/fs/file-max
> cat /proc/sys/fs/file-nr
>
>
>
> On Mon, May 5, 2014 at 10:31 PM, Bryan Talbot <bryan.talbot@playnext.com>wrote:
>
>> Running
>>
>> #> cat /proc/$(cat /var/run/cassandra.pid)/limits
>>
>> as root or your cassandra user will tell you what limits it's actually
>> running with.
>>
>>
>>
>>
>> On Sun, May 4, 2014 at 10:12 PM, Yatong Zhang <blueflycn@gmail.com>wrote:
>>
>>> I am running 'repair' when the error occurred. And just a few days
>>> before I changed the compaction strategy to 'leveled'. don know if this
>>> helps
>>>
>>>
>>> On Mon, May 5, 2014 at 1:10 PM, Yatong Zhang <blueflycn@gmail.com>wrote:
>>>
>>>> Cassandra is running as root
>>>>
>>>> [root@storage5 ~]# ps aux | grep java
>>>>> root      1893 42.0 24.0 7630664 3904000 ?     Sl   10:43  60:01 java
>>>>> -ea -javaagent:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar
>>>>> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
>>>>> -XX:ThreadPriorityPolicy=42 -Xms3959M -Xmx3959M -Xmn400M
>>>>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003
>>>>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>>>>> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
>>>>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>>>>> -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true
>>>>> -Dcom.sun.management.jmxremote.port=7199
>>>>> -Dcom.sun.management.jmxremote.ssl=false
>>>>> -Dcom.sun.management.jmxremote.authenticate=false
>>>>> -Dlog4j.configuration=log4j-server.properties
>>>>> -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/cassandra.pid
>>>>> -cp
>>>>> /mydb/cassandra/bin/../conf:/mydb/cassandra/bin/../build/classes/main:/mydb/cassandra/bin/../build/classes/thrift:/mydb/cassandra/bin/../lib/antlr-3.2.jar:/mydb/cassandra/bin/../lib/apache-cassandra-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-clientutil-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-thrift-2.0.7.jar:/mydb/cassandra/bin/../lib/commons-cli-1.1.jar:/mydb/cassandra/bin/../lib/commons-codec-1.2.jar:/mydb/cassandra/bin/../lib/commons-lang3-3.1.jar:/mydb/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/mydb/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/mydb/cassandra/bin/../lib/disruptor-3.0.1.jar:/mydb/cassandra/bin/../lib/guava-15.0.jar:/mydb/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/mydb/cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar:/mydb/cassandra/bin/../lib/jbcrypt-0.3m.jar:/mydb/cassandra/bin/../lib/jline-1.0.jar:/mydb/cassandra/bin/../lib/json-simple-1.1.jar:/mydb/cassandra/bin/../lib/libthrift-0.9.1.jar:/mydb/cassandra/bin/../lib/log4j-1.2.16.jar:/mydb/cassandra/bin/../lib/lz4-1.2.0.jar:/mydb/cassandra/bin/../lib/metrics-core-2.2.0.jar:/mydb/cassandra/bin/../lib/netty-3.6.6.Final.jar:/mydb/cassandra/bin/../lib/reporter-config-2.1.0.jar:/mydb/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/mydb/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/mydb/cassandra/bin/../lib/slf4j-log4j12-1.7.2.jar:/mydb/cassandra/bin/../lib/snakeyaml-1.11.jar:/mydb/cassandra/bin/../lib/snappy-java-1.0.5.jar:/mydb/cassandra/bin/../lib/snaptree-0.1.jar:/mydb/cassandra/bin/../lib/super-csv-2.1.0.jar:/mydb/cassandra/bin/../lib/thrift-server-0.3.3.jar
>>>>> org.apache.cassandra.service.CassandraDaemon
>>>>>
>>>>
>>>>
>>>>
>>>> On Mon, May 5, 2014 at 1:02 PM, Philip Persad <philip.persad@gmail.com>wrote:
>>>>
>>>>> Have you tried running "ulimit -a" as the Cassandra user instead of
>>>>> as root? It is possible that your configured a high file limit for root
but
>>>>> not for the user running the Cassandra process.
>>>>>
>>>>>
>>>>> On Sun, May 4, 2014 at 6:07 PM, Yatong Zhang <blueflycn@gmail.com>wrote:
>>>>>
>>>>>> [root@storage5 ~]# lsof -n | grep java | wc -l
>>>>>>> 5103
>>>>>>> [root@storage5 ~]# lsof | wc -l
>>>>>>> 6567
>>>>>>
>>>>>>
>>>>>> It's mentioned in previous mail:)
>>>>>>
>>>>>>
>>>>>> On Mon, May 5, 2014 at 9:03 AM, nash <nashef@gmail.com> wrote:
>>>>>>
>>>>>>> The lsof command or /proc can tell you how many open files it
has.
>>>>>>> How many is it?
>>>>>>>
>>>>>>> --nash
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>

Mime
View raw message