Return-Path: 
 <user-return-22000-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org>
X-Original-To: apmail-cassandra-user-archive@www.apache.org
Delivered-To: apmail-cassandra-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BB58E7A1F
	for <apmail-cassandra-user-archive@www.apache.org>;
 Thu,  3 Nov 2011 22:34:25 +0000 (UTC)
Received: (qmail 71507 invoked by uid 500); 3 Nov 2011 22:34:23 -0000
Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org
Received: (qmail 71474 invoked by uid 500); 3 Nov 2011 22:34:23 -0000
Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@cassandra.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@cassandra.apache.org>
List-Post: <mailto:user@cassandra.apache.org>
List-Id: <user.cassandra.apache.org>
Reply-To: user@cassandra.apache.org
Delivered-To: mailing list user@cassandra.apache.org
Received: (qmail 71466 invoked by uid 99); 3 Nov 2011 22:34:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Nov 2011 22:34:23 +0000
X-ASF-Spam-Status: No, hits=2.4 required=5.0
	tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.44] (HELO mail-yw0-f44.google.com) (209.85.213.44)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Nov 2011 22:34:14 +0000
Received: by ywt2 with SMTP id 2so2081488ywt.31
        for <user@cassandra.apache.org>; Thu, 03 Nov 2011 15:33:50 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.146.181.25 with SMTP id d25mr3068965yaf.3.1320359630286; Thu,
 03 Nov 2011 15:33:50 -0700 (PDT)
Received: by 10.147.32.5 with HTTP; Thu, 3 Nov 2011 15:33:50 -0700 (PDT)
X-Originating-IP: [173.167.104.65]
Date: Thu, 3 Nov 2011 15:33:50 -0700
Message-ID: 
 <CACE81Mij38TTr-Bp_QjcigNy-HHKYmo8D4Yz3QJof=fWOsGE5w@mail.gmail.com>
Subject: Read perf investigation
From: Ian Danforth <idanforth@numenta.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=000e0cd52b44bd31b004b0dc2ef0
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd52b44bd31b004b0dc2ef0
Content-Type: text/plain; charset=ISO-8859-1

All,

 I've done a bit more homework, and I continue to see long 200ms to 300ms
read times for some keys.

Test Setup

EC2 M1Large sending requests to a 5 node C* cluster also in EC2, also all
M1Large. RF=3. ReadConsistency = ONE. I'm using pycassa from python for all
communication.

Data Model

One column family with tens of millions of rows. The number of columns per
row varies between 0 and 1440 (per minute records). The values are all
ints. All data stored on EBS volumes. Total load per node is ~110GB.

According to VMstat I'm not swapping at all.

Highest %Util I see
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
xvdf              0.00  2788.00   17.00  267.50  1168.00 23020.00    85.02
   32.37  107.73   1.22  34.60

A more average profile I see is:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
xvdf              0.00     0.00   21.00    0.00  1288.00     0.00    61.33
    0.37   18.38   9.43  19.80

QUESTION

Where should I look next? I'd love to get a profile of exactly where
cassandra is spending its time on a per call basis.

Thanks in advance,

Ian

--000e0cd52b44bd31b004b0dc2ef0
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

All,<div><br></div><div>=A0I&#39;ve done a bit more homework, and I continu=
e to see long 200ms to 300ms read times for some keys.</div><div><br></div>=
<div>Test Setup</div><div><br></div><div>EC2 M1Large sending requests to a =
5 node C* cluster also in EC2, also all M1Large. RF=3D3. ReadConsistency =
=3D ONE. I&#39;m using pycassa from python for all communication.</div>
<div><br></div><div>Data Model</div><div><br></div><div>One column family w=
ith tens of millions of rows. The number of columns per row varies between =
0 and 1440 (per minute records). The values are all ints. All data stored o=
n EBS volumes. Total load per node is ~110GB.</div>
<div><br></div>According to VMstat I&#39;m not swapping at all.<div><br></d=
iv><div>Highest %Util I see</div><div><div><font class=3D"Apple-style-span"=
 face=3D"&#39;courier new&#39;, monospace">Device: =A0 =A0 =A0 =A0 rrqm/s =
=A0 wrqm/s =A0 =A0 r/s =A0 =A0 w/s =A0 rsec/s =A0 wsec/s avgrq-sz avgqu-sz =
=A0 await =A0svctm =A0%util</font></div>
<div><font class=3D"Apple-style-span" face=3D"&#39;courier new&#39;, monosp=
ace">xvdf =A0 =A0 =A0 =A0 =A0 =A0 =A00.00 =A02788.00 =A0 17.00 =A0267.50 =
=A01168.00 23020.00 =A0 =A085.02 =A0 =A032.37 =A0107.73 =A0 1.22 =A034.60</=
font></div></div><div><font class=3D"Apple-style-span" face=3D"&#39;courier=
 new&#39;, monospace"><br>
</font></div>A more average profile I see is:<div><font class=3D"Apple-styl=
e-span" face=3D"&#39;courier new&#39;, monospace"><br></font></div><div><fo=
nt class=3D"Apple-style-span" face=3D"&#39;courier new&#39;, monospace"><di=
v>Device: =A0 =A0 =A0 =A0 rrqm/s =A0 wrqm/s =A0 =A0 r/s =A0 =A0 w/s =A0 rse=
c/s =A0 wsec/s avgrq-sz avgqu-sz =A0 await =A0svctm =A0%util</div>
<div>xvdf =A0 =A0 =A0 =A0 =A0 =A0 =A00.00 =A0 =A0 0.00 =A0 21.00 =A0 =A00.0=
0 =A01288.00 =A0 =A0 0.00 =A0 =A061.33 =A0 =A0 0.37 =A0 18.38 =A0 9.43 =A01=
9.80</div></font></div><div><font class=3D"Apple-style-span" face=3D"&#39;c=
ourier new&#39;, monospace"><br></font></div>
QUESTION<div><br></div><div>Where should I look next? I&#39;d love to get a=
 profile of exactly where cassandra is spending its time on a per call basi=
s.</div><div><br></div><div>Thanks in advance,</div><div><br></div><div>
Ian</div>

--000e0cd52b44bd31b004b0dc2ef0--