Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 920AC118E4 for ; Fri, 16 May 2014 16:04:39 +0000 (UTC) Received: (qmail 78726 invoked by uid 500); 16 May 2014 14:23:34 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 34239 invoked by uid 500); 16 May 2014 13:56:34 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 46612 invoked by uid 99); 16 May 2014 13:49:28 -0000 Received: from Unknown (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 May 2014 13:49:28 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of blueflycn@gmail.com designates 209.85.214.177 as permitted sender) Received: from [209.85.214.177] (HELO mail-ob0-f177.google.com) (209.85.214.177) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 May 2014 13:42:04 +0000 Received: by mail-ob0-f177.google.com with SMTP id wp4so2870449obc.22 for ; Fri, 16 May 2014 06:41:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=f315s31nGdw/owhFHvgzbop2/+Orv6aSR2iawpBvvRw=; b=LaQ2rLQoD9a0qaEzEbTdanR16FLNefop19i7OImBVvUPQ0r89zgnmYnP88ZX/uPZ1s a/1mMLrtJvgvlxmEcBhNrflKkzLDaLnChv/0MS58nRmd5gVhYqgD61rnmhk8cYJecZ5c o+KnsAMTpQXS7sAR2fWQDxNzVu/QUAESN421ChTvV8bvGgPhMwLhWJthtOeG2bhUvyIf T+y1UGTmOLbouv2fFJ+Z6cfwXAyiBeA1OXQ90Cm6eO1Y+ekUEv7ODrG2LPnSIP5l/tGu R8++qrWoS1rdiZT3IUczXQSopA7KmeVBq53i3V+q977RMs7p0FfIbhr9yfdW00R8Qf3Q klmg== MIME-Version: 1.0 X-Received: by 10.60.92.170 with SMTP id cn10mr3071765oeb.76.1400247700490; Fri, 16 May 2014 06:41:40 -0700 (PDT) Received: by 10.182.24.201 with HTTP; Fri, 16 May 2014 06:41:40 -0700 (PDT) In-Reply-To: References: Date: Fri, 16 May 2014 21:41:40 +0800 Message-ID: Subject: Re: Cassandra 2.0.7 always failes due to 'too may open files' error From: Yatong Zhang To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b33d31ec90e9c04f984922a X-Virus-Checked: Checked by ClamAV on apache.org --047d7b33d31ec90e9c04f984922a Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Yes the global limits are OK. I added cassandra to '/etc/rc.local' to make it auto-startup, but seems the modification of limits didn't take effect. I observed this as Bryan suggested, so I added ulimit -SHn 999999 > to '/etc/rc.local' and before cassandra start command, and it worked. On Thu, May 8, 2014 at 3:34 AM, Nikolay Mihaylov wrote: > sorry, probably somebody mentioned it, but did you checked global limit? > > cat /proc/sys/fs/file-max > cat /proc/sys/fs/file-nr > > > > On Mon, May 5, 2014 at 10:31 PM, Bryan Talbot = wrote: > >> Running >> >> #> cat /proc/$(cat /var/run/cassandra.pid)/limits >> >> as root or your cassandra user will tell you what limits it's actually >> running with. >> >> >> >> >> On Sun, May 4, 2014 at 10:12 PM, Yatong Zhang wrote= : >> >>> I am running 'repair' when the error occurred. And just a few days >>> before I changed the compaction strategy to 'leveled'. don know if this >>> helps >>> >>> >>> On Mon, May 5, 2014 at 1:10 PM, Yatong Zhang wrote= : >>> >>>> Cassandra is running as root >>>> >>>> [root@storage5 ~]# ps aux | grep java >>>>> root 1893 42.0 24.0 7630664 3904000 ? Sl 10:43 60:01 java >>>>> -ea -javaagent:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar >>>>> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities >>>>> -XX:ThreadPriorityPolicy=3D42 -Xms3959M -Xmx3959M -Xmn400M >>>>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=3D100000= 3 >>>>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnable= d >>>>> -XX:SurvivorRatio=3D8 -XX:MaxTenuringThreshold=3D1 >>>>> -XX:CMSInitiatingOccupancyFraction=3D75 -XX:+UseCMSInitiatingOccupanc= yOnly >>>>> -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=3Dtrue >>>>> -Dcom.sun.management.jmxremote.port=3D7199 >>>>> -Dcom.sun.management.jmxremote.ssl=3Dfalse >>>>> -Dcom.sun.management.jmxremote.authenticate=3Dfalse >>>>> -Dlog4j.configuration=3Dlog4j-server.properties >>>>> -Dlog4j.defaultInitOverride=3Dtrue -Dcassandra-pidfile=3D/var/run/cas= sandra.pid >>>>> -cp >>>>> /mydb/cassandra/bin/../conf:/mydb/cassandra/bin/../build/classes/main= :/mydb/cassandra/bin/../build/classes/thrift:/mydb/cassandra/bin/../lib/ant= lr-3.2.jar:/mydb/cassandra/bin/../lib/apache-cassandra-2.0.7.jar:/mydb/cass= andra/bin/../lib/apache-cassandra-clientutil-2.0.7.jar:/mydb/cassandra/bin/= ../lib/apache-cassandra-thrift-2.0.7.jar:/mydb/cassandra/bin/../lib/commons= -cli-1.1.jar:/mydb/cassandra/bin/../lib/commons-codec-1.2.jar:/mydb/cassand= ra/bin/../lib/commons-lang3-3.1.jar:/mydb/cassandra/bin/../lib/compress-lzf= -0.8.4.jar:/mydb/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/= mydb/cassandra/bin/../lib/disruptor-3.0.1.jar:/mydb/cassandra/bin/../lib/gu= ava-15.0.jar:/mydb/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/mydb/cass= andra/bin/../lib/jackson-core-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jack= son-mapper-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar:/mydb/ca= ssandra/bin/../lib/jbcrypt-0.3m.jar:/mydb/cassandra/bin/../lib/jline-1.0.ja= r:/mydb/cassandra/bin/../lib/json-simple-1.1.jar:/mydb/cassandra/bin/../lib= /libthrift-0.9.1.jar:/mydb/cassandra/bin/../lib/log4j-1.2.16.jar:/mydb/cass= andra/bin/../lib/lz4-1.2.0.jar:/mydb/cassandra/bin/../lib/metrics-core-2.2.= 0.jar:/mydb/cassandra/bin/../lib/netty-3.6.6.Final.jar:/mydb/cassandra/bin/= ../lib/reporter-config-2.1.0.jar:/mydb/cassandra/bin/../lib/servlet-api-2.5= -20081211.jar:/mydb/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/mydb/cassandr= a/bin/../lib/slf4j-log4j12-1.7.2.jar:/mydb/cassandra/bin/../lib/snakeyaml-1= .11.jar:/mydb/cassandra/bin/../lib/snappy-java-1.0.5.jar:/mydb/cassandra/bi= n/../lib/snaptree-0.1.jar:/mydb/cassandra/bin/../lib/super-csv-2.1.0.jar:/m= ydb/cassandra/bin/../lib/thrift-server-0.3.3.jar >>>>> org.apache.cassandra.service.CassandraDaemon >>>>> >>>> >>>> >>>> >>>> On Mon, May 5, 2014 at 1:02 PM, Philip Persad wrote: >>>> >>>>> Have you tried running "ulimit -a" as the Cassandra user instead of >>>>> as root? It is possible that your configured a high file limit for ro= ot but >>>>> not for the user running the Cassandra process. >>>>> >>>>> >>>>> On Sun, May 4, 2014 at 6:07 PM, Yatong Zhang wro= te: >>>>> >>>>>> [root@storage5 ~]# lsof -n | grep java | wc -l >>>>>>> 5103 >>>>>>> [root@storage5 ~]# lsof | wc -l >>>>>>> 6567 >>>>>> >>>>>> >>>>>> It's mentioned in previous mail:) >>>>>> >>>>>> >>>>>> On Mon, May 5, 2014 at 9:03 AM, nash wrote: >>>>>> >>>>>>> The lsof command or /proc can tell you how many open files it has. >>>>>>> How many is it? >>>>>>> >>>>>>> --nash >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >> > --047d7b33d31ec90e9c04f984922a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Yes the global limits are OK. I added cassandra to '/e= tc/rc.local' to make it auto-startup, but seems the modification of lim= its didn't take effect. I observed this as Bryan suggested, so I added<= br>
ulimit -SHn 999999

to '/etc/rc.local' and before cas= sandra start command, and it worked.


O= n Thu, May 8, 2014 at 3:34 AM, Nikolay Mihaylov <nmmm@nmmm.nu> wr= ote:
sorry, probably somebody me= ntioned it, but did you checked global limit?

cat /= proc/sys/fs/file-max
cat /proc/sys/fs/file-nr



On Mon, May 5, 2014 at 10:31 PM, Bryan T= albot <bryan.talbot@playnext.com> wrote:
Running=C2=A0

#>=C2=A0cat /proc/$(ca= t /var/run/cassandra.pid)/limits

as root or your c= assandra user will tell you what limits it's actually running with.




On Sun, May 4, 2014 at 10:12 PM, Yatong Zhang <= ;blueflycn@gmail.c= om> wrote:
I am running 'repair= 9; when the error occurred. And just a few days before I changed the compac= tion strategy to 'leveled'. don know if this helps


On Mon, May 5, 2014 at 1:10 PM, Yatong Zhang <blueflycn@gmail.com&g= t; wrote:
Cassandra is running as root

[root@storage5 ~]# ps aux | grep java
root= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1893 42.0 24.0 7630664 3904000 ?=C2=A0=C2=A0= =C2=A0=C2=A0 Sl=C2=A0=C2=A0 10:43=C2=A0 60:01 java -ea -javaagent:/mydb/cas= sandra/bin/../lib/jamm-0.2.5.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThre= adPriorities -XX:ThreadPriorityPolicy=3D42 -Xms3959M -Xmx3959M -Xmn400M -XX= :+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=3D1000003 -XX:+Us= eParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:Survivo= rRatio=3D8 -XX:MaxTenuringThreshold=3D1 -XX:CMSInitiatingOccupancyFraction= =3D75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+UseCondCardMark = -Djava.net.preferIPv4Stack=3Dtrue -Dcom.sun.management.jmxremote.port=3D719= 9 -Dcom.sun.management.jmxremote.ssl=3Dfalse -Dcom.sun.management.jmxremote= .authenticate=3Dfalse -Dlog4j.configuration=3Dlog4j-server.properties -Dlog= 4j.defaultInitOverride=3Dtrue -Dcassandra-pidfile=3D/var/run/cassandra.pid = -cp /mydb/cassandra/bin/../conf:/mydb/cassandra/bin/../build/classes/main:/= mydb/cassandra/bin/../build/classes/thrift:/mydb/cassandra/bin/../lib/antlr= -3.2.jar:/mydb/cassandra/bin/../lib/apache-cassandra-2.0.7.jar:/mydb/cassan= dra/bin/../lib/apache-cassandra-clientutil-2.0.7.jar:/mydb/cassandra/bin/..= /lib/apache-cassandra-thrift-2.0.7.jar:/mydb/cassandra/bin/../lib/commons-c= li-1.1.jar:/mydb/cassandra/bin/../lib/commons-codec-1.2.jar:/mydb/cassandra= /bin/../lib/commons-lang3-3.1.jar:/mydb/cassandra/bin/../lib/compress-lzf-0= .8.4.jar:/mydb/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/my= db/cassandra/bin/../lib/disruptor-3.0.1.jar:/mydb/cassandra/bin/../lib/guav= a-15.0.jar:/mydb/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/mydb/cassan= dra/bin/../lib/jackson-core-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jackso= n-mapper-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar:/mydb/cass= andra/bin/../lib/jbcrypt-0.3m.jar:/mydb/cassandra/bin/../lib/jline-1.0.jar:= /mydb/cassandra/bin/../lib/json-simple-1.1.jar:/mydb/cassandra/bin/../lib/l= ibthrift-0.9.1.jar:/mydb/cassandra/bin/../lib/log4j-1.2.16.jar:/mydb/cassan= dra/bin/../lib/lz4-1.2.0.jar:/mydb/cassandra/bin/../lib/metrics-core-2.2.0.= jar:/mydb/cassandra/bin/../lib/netty-3.6.6.Final.jar:/mydb/cassandra/bin/..= /lib/reporter-config-2.1.0.jar:/mydb/cassandra/bin/../lib/servlet-api-2.5-2= 0081211.jar:/mydb/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/mydb/cassandra/= bin/../lib/slf4j-log4j12-1.7.2.jar:/mydb/cassandra/bin/../lib/snakeyaml-1.1= 1.jar:/mydb/cassandra/bin/../lib/snappy-java-1.0.5.jar:/mydb/cassandra/bin/= ../lib/snaptree-0.1.jar:/mydb/cassandra/bin/../lib/super-csv-2.1.0.jar:/myd= b/cassandra/bin/../lib/thrift-server-0.3.3.jar org.apache.cassandra.service= .CassandraDaemon



On Mon, May 5, 2014 at 1:02 PM, Philip Persad <p= hilip.persad@gmail.com> wrote:
Have you tried running &quo= t;

On Sun, May 4= , 2014 at 6:07 PM, Yatong Zhang <blueflycn@gmail.com> wrot= e:
[root@storage5 ~]# lsof -= n | grep java | wc -l
5103
[root@storage5 ~]# lsof | wc -l
6567
<= div>
It's mentioned in previous mail:)


On Mon, May 5, 2014 at 9:03 AM, nash <nashef@g= mail.com> wrote:
= The lsof command or /proc can tell you how many open files it has. How many= is it?

--nash








--047d7b33d31ec90e9c04f984922a--