Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CF79E116E4 for ; Mon, 5 May 2014 19:32:07 +0000 (UTC) Received: (qmail 41684 invoked by uid 500); 5 May 2014 19:32:01 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 41617 invoked by uid 500); 5 May 2014 19:32:01 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 41608 invoked by uid 99); 5 May 2014 19:32:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 May 2014 19:32:01 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: unknown (nike.apache.org: error in processing during lookup of bryan.talbot@playnext.com) Received: from [209.85.213.174] (HELO mail-ig0-f174.google.com) (209.85.213.174) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 May 2014 19:31:56 +0000 Received: by mail-ig0-f174.google.com with SMTP id h3so1919040igd.7 for ; Mon, 05 May 2014 12:31:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=t2bj9ktYa8lv+bVJsVqDeK1azltgimN/Qaxf2BFNWT0=; b=c5uQp4XS/qcWf2nANpPsQGu2CDESrJPdzH39mboY9NFSVd9UqCMu1R+japsD/oVux+ NDzBfLh6xufI5KKOxmz0aOGRoC5hVi6tg6GfYVY/t38PJYrvd37KtFwzoDBVSDdOYHBR /46lo+7wzi8Sz+3r6TetAoQznM+QI3KDXaJYAkU0nTcDEPCcleD/ne4oAX/v2hMPTKdG R6YDMmggv0QpO/2sLxgY4nE0wBLlmhI8+8tigD0kEaHQzKd9e8Cj8VzV7WqsoPMOVw1W J+Jl9F4qIxY4VBsGGwMb9igGTlZOGB2ZkFTsjo6NRsoRKEVzLz5p67iVAq6wgSMVKObI 7H5g== X-Gm-Message-State: ALoCoQmKPvhR8Z90tT5upabfRyQx66Ult1Cj/R8bJdaKWLjD6iu3tXrOY49/GzLQ6jGkXcTAoRNf MIME-Version: 1.0 X-Received: by 10.42.129.9 with SMTP id o9mr35043346ics.38.1399318292875; Mon, 05 May 2014 12:31:32 -0700 (PDT) Received: by 10.42.128.19 with HTTP; Mon, 5 May 2014 12:31:32 -0700 (PDT) In-Reply-To: References: Date: Mon, 5 May 2014 12:31:32 -0700 Message-ID: Subject: Re: Cassandra 2.0.7 always failes due to 'too may open files' error From: Bryan Talbot To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=20cf3010eb89c65cae04f8ac2d3e X-Virus-Checked: Checked by ClamAV on apache.org --20cf3010eb89c65cae04f8ac2d3e Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Running #> cat /proc/$(cat /var/run/cassandra.pid)/limits as root or your cassandra user will tell you what limits it's actually running with. On Sun, May 4, 2014 at 10:12 PM, Yatong Zhang wrote: > I am running 'repair' when the error occurred. And just a few days before > I changed the compaction strategy to 'leveled'. don know if this helps > > > On Mon, May 5, 2014 at 1:10 PM, Yatong Zhang wrote: > >> Cassandra is running as root >> >> [root@storage5 ~]# ps aux | grep java >>> root 1893 42.0 24.0 7630664 3904000 ? Sl 10:43 60:01 java >>> -ea -javaagent:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar >>> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities >>> -XX:ThreadPriorityPolicy=3D42 -Xms3959M -Xmx3959M -Xmn400M >>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=3D1000003 >>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled >>> -XX:SurvivorRatio=3D8 -XX:MaxTenuringThreshold=3D1 >>> -XX:CMSInitiatingOccupancyFraction=3D75 -XX:+UseCMSInitiatingOccupancyO= nly >>> -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=3Dtrue >>> -Dcom.sun.management.jmxremote.port=3D7199 >>> -Dcom.sun.management.jmxremote.ssl=3Dfalse >>> -Dcom.sun.management.jmxremote.authenticate=3Dfalse >>> -Dlog4j.configuration=3Dlog4j-server.properties >>> -Dlog4j.defaultInitOverride=3Dtrue -Dcassandra-pidfile=3D/var/run/cassa= ndra.pid >>> -cp >>> /mydb/cassandra/bin/../conf:/mydb/cassandra/bin/../build/classes/main:/= mydb/cassandra/bin/../build/classes/thrift:/mydb/cassandra/bin/../lib/antlr= -3.2.jar:/mydb/cassandra/bin/../lib/apache-cassandra-2.0.7.jar:/mydb/cassan= dra/bin/../lib/apache-cassandra-clientutil-2.0.7.jar:/mydb/cassandra/bin/..= /lib/apache-cassandra-thrift-2.0.7.jar:/mydb/cassandra/bin/../lib/commons-c= li-1.1.jar:/mydb/cassandra/bin/../lib/commons-codec-1.2.jar:/mydb/cassandra= /bin/../lib/commons-lang3-3.1.jar:/mydb/cassandra/bin/../lib/compress-lzf-0= .8.4.jar:/mydb/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/my= db/cassandra/bin/../lib/disruptor-3.0.1.jar:/mydb/cassandra/bin/../lib/guav= a-15.0.jar:/mydb/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/mydb/cassan= dra/bin/../lib/jackson-core-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jackso= n-mapper-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar:/mydb/cass= andra/bin/../lib/jbcrypt-0.3m.jar:/mydb/cassandra/bin/../lib/jline-1.0.jar:= /mydb/cassandra/bin/../lib/json-simple-1.1.jar:/mydb/cassandra/bin/../lib/l= ibthrift-0.9.1.jar:/mydb/cassandra/bin/../lib/log4j-1.2.16.jar:/mydb/cassan= dra/bin/../lib/lz4-1.2.0.jar:/mydb/cassandra/bin/../lib/metrics-core-2.2.0.= jar:/mydb/cassandra/bin/../lib/netty-3.6.6.Final.jar:/mydb/cassandra/bin/..= /lib/reporter-config-2.1.0.jar:/mydb/cassandra/bin/../lib/servlet-api-2.5-2= 0081211.jar:/mydb/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/mydb/cassandra/= bin/../lib/slf4j-log4j12-1.7.2.jar:/mydb/cassandra/bin/../lib/snakeyaml-1.1= 1.jar:/mydb/cassandra/bin/../lib/snappy-java-1.0.5.jar:/mydb/cassandra/bin/= ../lib/snaptree-0.1.jar:/mydb/cassandra/bin/../lib/super-csv-2.1.0.jar:/myd= b/cassandra/bin/../lib/thrift-server-0.3.3.jar >>> org.apache.cassandra.service.CassandraDaemon >>> >> >> >> >> On Mon, May 5, 2014 at 1:02 PM, Philip Persad w= rote: >> >>> Have you tried running "ulimit -a" as the Cassandra user instead of as >>> root? It is possible that your configured a high file limit for root bu= t >>> not for the user running the Cassandra process. >>> >>> >>> On Sun, May 4, 2014 at 6:07 PM, Yatong Zhang wrote= : >>> >>>> [root@storage5 ~]# lsof -n | grep java | wc -l >>>>> 5103 >>>>> [root@storage5 ~]# lsof | wc -l >>>>> 6567 >>>> >>>> >>>> It's mentioned in previous mail:) >>>> >>>> >>>> On Mon, May 5, 2014 at 9:03 AM, nash wrote: >>>> >>>>> The lsof command or /proc can tell you how many open files it has. Ho= w >>>>> many is it? >>>>> >>>>> --nash >>>>> >>>> >>>> >>> >> > --20cf3010eb89c65cae04f8ac2d3e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Running=C2=A0

#>=C2=A0cat /proc/$(ca= t /var/run/cassandra.pid)/limits

as root or your c= assandra user will tell you what limits it's actually running with.




On Sun, May 4, 2014 at 10:12 PM, Yatong Zhang <= ;blueflycn@gmail.c= om> wrote:
I am running 'repair= 9; when the error occurred. And just a few days before I changed the compac= tion strategy to 'leveled'. don know if this helps

On Mon, May 5, 2014 at 1:10 PM, Yatong Zhang <blueflycn@gmail.com&g= t; wrote:
Cassandra is running as root

[root@storage5 ~]# ps aux | grep java
root= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1893 42.0 24.0 7630664 3904000 ?=C2=A0=C2=A0= =C2=A0=C2=A0 Sl=C2=A0=C2=A0 10:43=C2=A0 60:01 java -ea -javaagent:/mydb/cas= sandra/bin/../lib/jamm-0.2.5.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThre= adPriorities -XX:ThreadPriorityPolicy=3D42 -Xms3959M -Xmx3959M -Xmn400M -XX= :+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=3D1000003 -XX:+Us= eParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:Survivo= rRatio=3D8 -XX:MaxTenuringThreshold=3D1 -XX:CMSInitiatingOccupancyFraction= =3D75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+UseCondCardMark = -Djava.net.preferIPv4Stack=3Dtrue -Dcom.sun.management.jmxremote.port=3D719= 9 -Dcom.sun.management.jmxremote.ssl=3Dfalse -Dcom.sun.management.jmxremote= .authenticate=3Dfalse -Dlog4j.configuration=3Dlog4j-server.properties -Dlog= 4j.defaultInitOverride=3Dtrue -Dcassandra-pidfile=3D/var/run/cassandra.pid = -cp /mydb/cassandra/bin/../conf:/mydb/cassandra/bin/../build/classes/main:/= mydb/cassandra/bin/../build/classes/thrift:/mydb/cassandra/bin/../lib/antlr= -3.2.jar:/mydb/cassandra/bin/../lib/apache-cassandra-2.0.7.jar:/mydb/cassan= dra/bin/../lib/apache-cassandra-clientutil-2.0.7.jar:/mydb/cassandra/bin/..= /lib/apache-cassandra-thrift-2.0.7.jar:/mydb/cassandra/bin/../lib/commons-c= li-1.1.jar:/mydb/cassandra/bin/../lib/commons-codec-1.2.jar:/mydb/cassandra= /bin/../lib/commons-lang3-3.1.jar:/mydb/cassandra/bin/../lib/compress-lzf-0= .8.4.jar:/mydb/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/my= db/cassandra/bin/../lib/disruptor-3.0.1.jar:/mydb/cassandra/bin/../lib/guav= a-15.0.jar:/mydb/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/mydb/cassan= dra/bin/../lib/jackson-core-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jackso= n-mapper-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar:/mydb/cass= andra/bin/../lib/jbcrypt-0.3m.jar:/mydb/cassandra/bin/../lib/jline-1.0.jar:= /mydb/cassandra/bin/../lib/json-simple-1.1.jar:/mydb/cassandra/bin/../lib/l= ibthrift-0.9.1.jar:/mydb/cassandra/bin/../lib/log4j-1.2.16.jar:/mydb/cassan= dra/bin/../lib/lz4-1.2.0.jar:/mydb/cassandra/bin/../lib/metrics-core-2.2.0.= jar:/mydb/cassandra/bin/../lib/netty-3.6.6.Final.jar:/mydb/cassandra/bin/..= /lib/reporter-config-2.1.0.jar:/mydb/cassandra/bin/../lib/servlet-api-2.5-2= 0081211.jar:/mydb/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/mydb/cassandra/= bin/../lib/slf4j-log4j12-1.7.2.jar:/mydb/cassandra/bin/../lib/snakeyaml-1.1= 1.jar:/mydb/cassandra/bin/../lib/snappy-java-1.0.5.jar:/mydb/cassandra/bin/= ../lib/snaptree-0.1.jar:/mydb/cassandra/bin/../lib/super-csv-2.1.0.jar:/myd= b/cassandra/bin/../lib/thrift-server-0.3.3.jar org.apache.cassandra.service= .CassandraDaemon



On Mon, May 5, 2014 at 1:02 PM, Philip Persad <p= hilip.persad@gmail.com> wrote:
Have you tried running &quo= t;

On Sun, May 4= , 2014 at 6:07 PM, Yatong Zhang <blueflycn@gmail.com> wrot= e:
[root@storage5 ~]# lsof -= n | grep java | wc -l
5103
[root@storage5 ~]# lsof | wc -l
6567
<= div>
It's mentioned in previous mail:)


On Mon, May 5, 2014 at 9:03 AM, nash <nashef@g= mail.com> wrote:
= The lsof command or /proc can tell you how many open files it has. How many= is it?

--nash






--20cf3010eb89c65cae04f8ac2d3e--