Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 70388 invoked from network); 22 Mar 2010 22:14:42 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 22 Mar 2010 22:14:42 -0000 Received: (qmail 15773 invoked by uid 500); 22 Mar 2010 22:14:41 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 15761 invoked by uid 500); 22 Mar 2010 22:14:41 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 15753 invoked by uid 99); 22 Mar 2010 22:14:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 22:14:41 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of bburruss@real.com designates 207.188.23.4 as permitted sender) Received: from [207.188.23.4] (HELO kal-el.real.com) (207.188.23.4) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 22:14:32 +0000 Received: from seacas02.corp.real.com ([::ffff:192.168.139.57]) (TLS: TLSv1/SSLv3,128bits,AES128-SHA) by kal-el.real.com with esmtp; Mon, 22 Mar 2010 15:14:10 -0700 id 0008000B.4BA7EBB2.0000373C Received: from seambx.corp.real.com ([fe80::2d15:fda7:b3b8:e268]) by seacas02.corp.real.com ([::1]) with mapi; Mon, 22 Mar 2010 15:14:09 -0700 From: Todd Burruss To: "user@cassandra.apache.org" Date: Mon, 22 Mar 2010 15:14:09 -0700 Subject: memory question Thread-Topic: memory question Thread-Index: AQHKygv3/HQU0Pe4kUW4prz0hpTdBQ== Message-ID: <766B5A29D28DA442AB229AAEE2AFC44507DF67ABC8@SEAMBX.corp.real.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_766B5A29D28DA442AB229AAEE2AFC44507DF67ABC8SEAMBXcorprea_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_766B5A29D28DA442AB229AAEE2AFC44507DF67ABC8SEAMBXcorprea_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable after running my cluster for a while performance has become unacceptable, 2= 00+ ms for reads. if running well, i see reads <10ms. when i run iostat t= he disk is being hammered by reads. seems like i/o caching isn't even bein= g used avg-cpu: %user %nice %system %iowait %steal %idle 2.81 0.00 1.41 13.62 0.00 82.16 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgq= u-sz await svctm %util cciss/c0d0p1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0= 0 0.00 0.00 0.00 cciss/c0d1p1 0.00 0.00 848.50 0.00 13.66 0.00 32.98 21.= 50 25.23 1.18 100.05 i run top and i see cassandra's memory usage as follows: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31510 bburruss 19 0 359g 37g 27g S 48.8 80.1 2137:30 java i set -Xmx10g so it isn't java using the memory. is it mmap i/o? what wou= ld be causing the huge memory usage? it seems reasonable that the performa= nce is bad because the i/o cache can't be used properly. --_000_766B5A29D28DA442AB229AAEE2AFC44507DF67ABC8SEAMBXcorprea_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

after = running my cluster for a while performance has become unacceptable, 200+= ; ms for reads. if running well, i see reads <10ms. when i run iostat the disk is= being hammered by reads. seems like i/o caching isn't even being use= d

avg-cpu: %user %nice %system %iowait&nbs= p; %steal %idle
2.81 &nbs= p; 0.00 1.41 13.62 0.= 00 82.16

Device:         rr= qm/s   wrqm/s   r/s   w/s    r= MB/s    wMB/s avgrq-sz avgqu-sz   await svct= m %util
cciss/c0d0p1
            &nb= sp; 0.00     0.00 0.00 0.00 &nbs= p;   0.00     0.00     0.= 00     0.00    0.00   0.00&nbs= p; 0.00
cciss/c0d1p1
            &nb= sp; 0.00     0.00 848.50 0.00  &n= bsp; 13.66     0.00    32.98  = 21.50   25.23   1.18 100.05

i run top and i see cass= andra's memory usage as follows:

PID USER&nbs= p; PR NI VIRT RES SHR S %CPU %MEM = TIME+ COMMAND

31510 bburruss 19 = 0 359g 37g 27g S 48.8 80.1 2137:30 java

i set -Xmx10g so it isn'= t java using the memory. is it mmap i/o? what would be c= ausing the huge memory usage? it seems reasonable that the perfo= rmance is bad because the i/o cache can't be used properly.

--_000_766B5A29D28DA442AB229AAEE2AFC44507DF67ABC8SEAMBXcorprea_--