Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5B6BD10140 for ; Thu, 7 Nov 2013 11:46:35 +0000 (UTC) Received: (qmail 73481 invoked by uid 500); 7 Nov 2013 11:46:31 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 73412 invoked by uid 500); 7 Nov 2013 11:46:29 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 73404 invoked by uid 99); 7 Nov 2013 11:46:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Nov 2013 11:46:28 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kmurthy7@gmail.com designates 209.85.212.172 as permitted sender) Received: from [209.85.212.172] (HELO mail-wi0-f172.google.com) (209.85.212.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Nov 2013 11:46:22 +0000 Received: by mail-wi0-f172.google.com with SMTP id ez12so464608wid.17 for ; Thu, 07 Nov 2013 03:46:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=VlDSEmfnCHmkF0cl6a59z4GEFuu1U8Y++eHdYBIBUio=; b=Jl/Ne4FLqWTLc4GpE/LQfQxY9WPI4hcHxNDaDzPu/jGlV4TIimNi847YYJ6ALGKort VBFxLZ1StmNPJ/TXi6DPHoBL9hobpHtqHA3Zz+wzXAiFQNd1qlrAXCqqopaK2+pwdqTu TF1CG7kQ2ppEpnu5OU4TArWiKqDcFZF4iU7Jz5urG8WhwMivq+drRLYa6xdCyodIxHOf sx0MY2SRvslhb2b6bsBeX9ZrwLjzbnO8DUl8xvURFA0xGpUn+9IOANAoicbHVJDfY4lP UOBUx+2U/xNMNzBwAliauzCP0WBqt0Q8/i0y2yfACOYUf3xLMHXROCXv8oalGfEyXCUe Cy+w== MIME-Version: 1.0 X-Received: by 10.194.21.225 with SMTP id y1mr1352215wje.60.1383824762036; Thu, 07 Nov 2013 03:46:02 -0800 (PST) Received: by 10.194.44.73 with HTTP; Thu, 7 Nov 2013 03:46:01 -0800 (PST) In-Reply-To: <1c3e690c63154758be5bfa4fb48ae8f6@AMSPR06MB069.eurprd06.prod.outlook.com> References: <1c3e690c63154758be5bfa4fb48ae8f6@AMSPR06MB069.eurprd06.prod.outlook.com> Date: Thu, 7 Nov 2013 17:16:01 +0530 Message-ID: Subject: Re: Getting into Too many open files issues From: Murthy Chelankuri To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b5d98ab5f693904ea94cfdc X-Virus-Checked: Checked by ClamAV on apache.org --047d7b5d98ab5f693904ea94cfdc Content-Type: text/plain; charset=ISO-8859-1 Thanks Pieter for giving quick reply. I have downloaded the tar ball. And have changed the limits.conf as per the documentation like below. * soft nofile 32768 * hard nofile 32768 root soft nofile 32768 root hard nofile 32768 * soft memlock unlimited * hard memlock unlimited root soft memlock unlimited root hard memlock unlimited * soft as unlimited * hard as unlimited root soft as unlimited root hard as unlimited root soft/hard nproc 32000 Some reason with in less than an hour cassandra node is opening 32768 files and cassandra is not responding after that. It is still not clear why cassadra is opening that many files and not closing properly ( does the laest cassandra 2.0.1 version have some bugs ). what i have been experimenting is 300 writes per sec and 500 reads per sec. And i have using 2 node cluster with 8 core cpu and 32GB RAM ( Virtuval Machines) Do we need to increase the nofile limts to more than 32768 ? On Thu, Nov 7, 2013 at 4:55 PM, Pieter Callewaert < pieter.callewaert@be-mobile.be> wrote: > Hi Murthy, > > > > Did you do a package install (.deb?) or you downloaded the tar? > > If the latest, you have to adjust the limits.conf file > (/etc/security/limits.conf) to raise the nofile (number of files open) for > the cassandra user. > > > > If you are using the .deb package, the limit is already raised to 100 000 > files. (can be found in /etc/init.d/cassandra, FD_LIMIT). > > However, with the 2.0.x I had to raise it to 1 000 000 because 100 000 was > too low. > > > > Kind regards, > > Pieter Callewaert > > > > *From:* Murthy Chelankuri [mailto:kmurthy7@gmail.com] > *Sent:* donderdag 7 november 2013 12:15 > *To:* user@cassandra.apache.org > *Subject:* Getting into Too many open files issues > > > > I have experimenting cassandra latest version for storing the huge the in > our application. > > Write are doing good. but when comes to reads i have obsereved that > cassandra is getting into too many open files issues. When i check the logs > its not able to open the cassandra data files any more before of the file > descriptors limits. > > Can some one suggest me what i am going wrong what could be issues > which causing the read operating leads to Too many open files issue. > --047d7b5d98ab5f693904ea94cfdc Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks Pieter for giving quick reply.

I = have downloaded=A0 the tar ball. And have changed the limits.conf as per th= e documentation like below.

* soft nofile 32768
* hard= nofile 32768
root soft nofile 32768
root hard nofile 32768
* soft memlock unlimite= d
* hard memlock unlimited
root soft memlock unlimited
root hard m= emlock unlimited
* soft as unlimited
* hard as unlimited
root soft= as unlimited
root hard as unlimited

root soft/hard nproc 32000



Some reason with in less than an hour cassandra node is opening 327= 68 files and cassandra is not responding after that.

It is still not clear why cassadra is opening that many files and not closi= ng properly ( does the laest cassandra 2.0.1 version have some bugs ).
<= br>
what i have been experimenting is 300 writes per sec and 500 = reads per sec.

And i have using 2 node cluster with 8 core cpu and 32GB RAM= ( Virtuval Machines)


Do we need to increa= se the nofile limts to more than 32768 ?
















On Thu, Nov 7, 2013= at 4:55 PM, Pieter Callewaert <pieter.callewaert@be-mobile.b= e> wrote:

Hi Murthy,=

=A0=

Did you do= a package install (.deb?) or you downloaded the tar?<= /p>

If the lat= est, you have to adjust the limits.conf file (/etc/security/limits.conf) to= raise the nofile (number of files open) for the cassandra user.

=A0=

If you are= using the .deb package, the limit is already raised to 100 000 files. (can= be found in /etc/init.d/cassandra, FD_LIMIT).

However, w= ith the 2.0.x I had to raise it to 1 000 000 because 100 000 was too low.

=A0=

Kind regar= ds,

Pieter Cal= lewaert

=A0<= /p>

From: Murthy Chelankuri [mailto:kmurthy7@gmail.com]
Sent: donderdag 7 november 2013 12:15
To: u= ser@cassandra.apache.org
Subject: Getting into Too many open files issues

=A0

I have experimenting = cassandra latest version for storing the huge the in our application.

Write are doing good.= but when comes to reads i have obsereved that cassandra is getting into to= o many open files issues. When i check the logs its not able to open the ca= ssandra data files any more before of the file descriptors limits.

Can some one suggest me what i am going wrong what c= ould be issues which causing the read operating leads to Too many open file= s issue.


--047d7b5d98ab5f693904ea94cfdc--