Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cassandra-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 209.85.219.209 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=uWChYb/BsB9kCef92ESMW1LB67fMjEu8IHtL2tNgrBh+lw+iHtxuXpmbF6HvX13dq8
         ZgoBD+JjhBWqeH1OVl589alVMBc3cYTPyCBwHBWj5AS37MpD0ovptPmaH7CIINmtECm0
         3XhAsz7Qd6Lh+XByMRJ+uIOxAfJ4ZdIYkDxeQ=
MIME-Version: 1.0
In-Reply-To: <f26ac6db0908171214y325ba42apb034100806e405c1@mail.gmail.com>
References: <f26ac6db0908171214y325ba42apb034100806e405c1@mail.gmail.com>
Date: Mon, 17 Aug 2009 17:14:23 -0700
Message-ID: <e06563880908171714o1642fd55x79dddb47a93f7f38@mail.gmail.com>
Subject: Re: Cassandra performance
From: Jonathan Ellis <jbellis@gmail.com>
To: cassandra-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

What happened about 20h in to make the latency drop so dramatically?

On Mon, Aug 17, 2009 at 12:14 PM, Huming Wu<huming.wu@gmail.com> wrote:
> I did some performance test and I am not impressed :). The data set is
> 880K unique keys and there are 4 columns with 2 columns being string
> and the other 2 are integers (from client side, to the backend it is
> all byte[]). After high throughput set (very fast), 220MB are injected
> via batch_insert. I restarted the cassandra and started a client
> calling get_slice at 5000rps with 100 connections. Here are some
> graphs over 2 days:
>
> 1. rps/qps: =A0http://farm3.static.flickr.com/2585/3831093496_068b90caa0_=
o.png
> 2. latency: =A0http://farm4.static.flickr.com/3421/3830297179_8decd66e34_=
o.png
> 3. CPU: http://farm4.static.flickr.com/3432/3831093584_b5bd459f55_o.png
> 4. mem: http://farm4.static.flickr.com/3526/3830356879_d09ac2695c_o.png
>
> A couple of observations:
>
> a) Read is too CPU intensive. With the actual peak rps around 3000,
> the CPU usage is 70% already. I doubt it I can double the rps and have
> the same read latency.
> b) The memory footprint is too big given the data size. I used
> incremental QC. I am pretty new to JAVA especially for the performance
> tuning. So maybe something is not right in the setting. But here is
> the JVM config:
>
> -Xmx6000m -Xms6000m -XX:+HeapDumpOnOutOfMemoryError -XX:NewSize=3D1000m
> -XX:MaxNewSize=3D1000m -XX:SurvivorRatio=3D8 -XX:+UseConcMarkSweepGC
> -XX:+CMSIncrementalMode
>
> The machines are 8 cores and 8G RAM. =A0here are some configuration
> parameters (client is doing non block get_slice):
> =A0 =A0<ReplicationFactor>2</ReplicationFactor>
> =A0 =A0<MemtableSizeInMB>1024</MemtableSizeInMB>
> =A0 =A0<MemtableObjectCountInMillions>2</MemtableObjectCountInMillions>
> =A0 =A0<KeysCachedFraction>1</KeysCachedFraction>
> =A0 =A0<ConcurrentReads>8</ConcurrentReads>
> =A0 =A0<ConcurrentWrites>32</ConcurrentWrites>
>
> The performance is very important to us (under high throughput). I did
> some preliminary test on sustained put and get and the performance is
> worse. But I thought I started the report with read only first.
>
> Any comments on those numbers?
>
> Thanks,
> Huming
>
> p.s. I am using trunk as of Aug. 12
>
> svn info
> Path: .
> URL: https://svn.apache.org/repos/asf/incubator/cassandra/trunk
> Repository Root: https://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 803947
> Node Kind: directory
> Schedule: normal
> Last Changed Author: jbellis
> Last Changed Rev: 803716
> Last Changed Date: 2009-08-12 21:27:24 +0000 (Wed, 12 Aug 2009)
>