Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A7B2B220E for ; Thu, 5 May 2011 16:01:48 +0000 (UTC) Received: (qmail 25154 invoked by uid 500); 5 May 2011 16:01:46 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 25130 invoked by uid 500); 5 May 2011 16:01:46 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 25122 invoked by uid 99); 5 May 2011 16:01:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:01:46 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates 209.85.220.172 as permitted sender) Received: from [209.85.220.172] (HELO mail-vx0-f172.google.com) (209.85.220.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:01:42 +0000 Received: by vxg33 with SMTP id 33so2913837vxg.31 for ; Thu, 05 May 2011 09:01:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type:content-transfer-encoding; bh=D1hOEx20vGR4OaIDmd2NUcDrPDVuf+vBBkeHEmghrFA=; b=Dj+/uoeBcTPLTlwTLqNOPnbZmPgutdlEuIMFO5JqgtaxuwwD1Pih43Uol91iBUfG0T fD3QqJRKjyscJreL5yis86+MbOp9YKPmMyPD6DnW2ThKoTzd7jWUosNvS3eFI9rJwfkT iW1/AtMYRMsBV8cTNL0Pcd2RoUggBNOt58lSo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; b=hGliet56J8vCZkoSXpJW4Kpm8RkjgCFnF9XQNinkRIdyCz+pmUYrOZVzLwJfOYZZUk xnb56laFQx7H4bgzjkhga/+UhtEtBuvQGCzojqVh3UnIPNJ/MTkh7hQ5wng5eCn3hoCe +vaWNOSYhoWlACfuRWnvpPD/hobuOWdccELeg= Received: by 10.52.73.33 with SMTP id i1mr2071361vdv.133.1304611281100; Thu, 05 May 2011 09:01:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.184.65 with HTTP; Thu, 5 May 2011 09:01:01 -0700 (PDT) In-Reply-To: <4DC2C79D.4070703@panasiangroup.com> References: <4DC2C79D.4070703@panasiangroup.com> From: Jonathan Ellis Date: Thu, 5 May 2011 11:01:01 -0500 Message-ID: Subject: Re: GC for ParNew (cassandra under performance ) 0.6.3 To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable 6s parnew is insane. you're probably swapping. Easiest fix is disabling swap entirely. P.S. 0.6.3 is ancient. On Thu, May 5, 2011 at 10:51 AM, Ali Ahsan wr= ote: > Hi All > > I have two cassandra node with RF=3D2,Cassandra start under performing = =A0i mean > red write are slow and i see following info in cassandra log.I have cento= s > 5.5 64 bit with 14 GB memory assigned to cassandra.I am using LVM for > cassandra.When i reboot my cassanrda every thing become normal. > > Below is some nodetoll info > > > # /root/cassandra/bin/nodetool -h 10.0.0.4 info > 222289592293417191096473651402050455 > Load =A0 =A0 =A0 =A0 =A0 =A0 : 43.51 GB > Generation No =A0 =A0: 1304495724 > Uptime (seconds) : 114699 > Heap Memory (MB) : 9944.43 / 14319.38 > > > tpstats > > [root@cassandra2 cassandra]# /root/cassandra/bin/nodetool -h 10.0.0.4 > tpstats > Pool Name =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Active =A0 Pending =A0 = =A0 =A0Completed > FILEUTILS-DELETE-POOL =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 =A0= =A0 =A0 =A0 =A0187 > STREAM-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0= 0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 > RESPONSE-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0= =A0 =A0 =A0 35901940 > ROW-READ-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A08 =A0 =A0 =A0 =A0 9= =A0 =A0 =A0 14580549 > LB-OPERATIONS =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0= =A0 =A0 =A0 =A0 =A0 =A0 =A00 > MESSAGE-DESERIALIZER-POOL =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= 54586854 > GMFD =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0= =A0 =A0 0 =A0 =A0 =A0 =A0 308503 > LB-TARGET =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 > CONSISTENCY-MANAGER =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A07637656 > ROW-MUTATION-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A02 2 43329077 > MESSAGE-STREAMING-POOL =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 =A0 =A0= =A0 =A0 =A0 =A0 =A00 > LOAD-BALANCER-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 =A0 =A00 > FLUSH-SORTER-POOL =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0= =A0 =A0 =A0 =A0 =A0 =A00 > MEMTABLE-POST-FLUSHER =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 =A0= =A0 =A0 =A0 2028 > FLUSH-WRITER-POOL =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0= =A0 =A0 =A0 =A0 2028 > AE-SERVICE-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 = =A0 =A0 =A0 =A0 =A0 =A0 =A00 > HINTED-HANDOFF-POOL =A0 =A0 =A0 =A0 =A0 =A0 =A0 1 =A0 =A0 =A0 =A030 =A0 = =A0 =A0 =A0 =A0 =A0 97 > > > > When i see below info cassandra start under performing and happening on b= oth > machines. > > > INFO [GC inspection] 2011-05-05 20:39:08,617 GCInspector.java (line 110) = GC > for ParNew: 698 ms, 138706360 reclaimed leaving 10135848048 used; max is > 15241445376 > =A0INFO [GC inspection] 2011-05-05 20:39:09,620 GCInspector.java (line 11= 0) GC > for ParNew: 496 ms, 138960224 reclaimed leaving 10135360416 used; max is > 15241445376 > =A0INFO [GC inspection] 2011-05-05 20:39:10,988 GCInspector.java (line 11= 0) GC > for ParNew: 1051 ms, 137654632 reclaimed leaving 10137177208 used; max is > 15241445376 > =A0INFO [GC inspection] 2011-05-05 20:39:13,370 GCInspector.java (line 11= 0) GC > for ParNew: 907 ms, 137254552 reclaimed leaving 10137977152 used; max is > 15241445376 > =A0INFO [GC inspection] 2011-05-05 20:39:14,631 GCInspector.java (line 11= 0) GC > for ParNew: 635 ms, 135828080 reclaimed leaving 10140229096 used; max is > 15241445376 > =A0INFO [GC inspection] 2011-05-05 20:39:16,872 GCInspector.java (line 11= 0) GC > for ParNew: 1910 ms, 137722664 reclaimed leaving 10141548552 used; max is > 15241445376 > =A0INFO [GC inspection] 2011-05-05 20:39:23,463 GCInspector.java (line 11= 0) GC > for ParNew: 6341 ms, 139920392 reclaimed leaving 10141215712 used; max is > 15241445376 > =A0INFO [GC inspection] 2011-05-05 20:39:25,482 GCInspector.java (line 11= 0) GC > for ParNew: 1712 ms, 139482032 reclaimed leaving 10141321232 used; max is > 15241445376 > > > -- > S.Ali Ahsan > > Senior System Engineer > > e-Business (Pvt) Ltd > > 49-C Jail Road, Lahore, P.O. Box 676 > Lahore 54000, Pakistan > > Tel: +92 (0)42 3758 7140 Ext. 128 > > Mobile: +92 (0)345 831 8769 > > Fax: +92 (0)42 3758 0027 > > Email: ali.ahsan@panasiangroup.com > > > > www.ebusiness-pg.com > > www.panasiangroup.com > > Confidentiality: This e-mail and any attachments may be confidential > and/or privileged. If you are not a named recipient, please notify the > sender immediately and do not disclose the contents to another person > use it for any purpose or store or copy the information in any medium. > Internet communications cannot be guaranteed to be timely, secure, error > or virus-free. We do not accept liability for any errors or omissions. > > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com