Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 71797 invoked from network); 10 Mar 2011 16:09:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 10 Mar 2011 16:09:22 -0000 Received: (qmail 15797 invoked by uid 500); 10 Mar 2011 16:09:20 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 15758 invoked by uid 500); 10 Mar 2011 16:09:20 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 15750 invoked by uid 99); 10 Mar 2011 16:09:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 16:09:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sridhar.basam@gmail.com designates 209.85.213.172 as permitted sender) Received: from [209.85.213.172] (HELO mail-yx0-f172.google.com) (209.85.213.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 16:09:15 +0000 Received: by yxk30 with SMTP id 30so950528yxk.31 for ; Thu, 10 Mar 2011 08:08:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=4A+1VwfnDL/swbc6sIKYb15XWeTIf0lMNn+XTRoCCSk=; b=Lwhj4gjPfgduguq4O5iLLALmYgBJUXKhxrvqlU4YzacYHppPGi80wCCCbhGBaKGZU1 iubBfse3QXej9DSJ36T7beVYz4CQ+rWRivE3f8flnSUQjb1J9x3pCOqTno+sXHuPGhGG xaSwbxxGiJEa7xOMzBfe6b8nZ0lyi44HZZ2AE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=K298SukeQwsvavMaDmZM5KYPwMAgdgBonjvJ7OrSmiqMRRu6mp3IpndpAMY3iX47gA C7Kw6HmImjq8GqUOZnbDBK6A26uIh8JT5SRlauAYmVhPozrTG54ZFDjMFDmSWyZ1N8RK UHEs0qa+NYKyNil/026jjZnN9LW9RkflXdVHo= MIME-Version: 1.0 Received: by 10.52.88.144 with SMTP id bg16mr1535710vdb.27.1299773334635; Thu, 10 Mar 2011 08:08:54 -0800 (PST) Sender: sridhar.basam@gmail.com Received: by 10.220.182.8 with HTTP; Thu, 10 Mar 2011 08:08:54 -0800 (PST) In-Reply-To: References: Date: Thu, 10 Mar 2011 11:08:54 -0500 X-Google-Sender-Auth: Dqd6C4RhmEVApKtKr6aTyIsBuSQ Message-ID: Subject: Re: mutator.execute() timings - big variance noted - pointers needed on understanding/improving it From: sridhar basam To: user@cassandra.apache.org Cc: Roshan Dawrani , hector-users@googlegroups.com Content-Type: multipart/alternative; boundary=20cf307d01c2e64545049e230fff --20cf307d01c2e64545049e230fff Content-Type: text/plain; charset=ISO-8859-1 Sounds like GC from your description of fast->slow->fast. Collect GC times from both the client and server side and plot against your application timing. If you uncomment the verbose GC entries in the cassandra-env.sh file you should get timing for the server side, pass in the same arguments for your client. Align time across the 3 files and plot to see if GC is the cause. Sridhar On Thu, Mar 10, 2011 at 9:30 AM, Roshan Dawrani wrote: > Hi, > > I am in the middle of some load testing on a 1-node Cassandra setup. We are > not on very high loads yet. We have recorded the timings taken up by > mutator.execute() calls and we see this kind of variation during the test > run: > > So, 25% of the times, execute() calls come back in 25 milli-seconds, but > the longer calls go upto 4 seconds. > > Can someone please provide some pointers on what and where to focus on in > my Hector / Cassandra setup? We are mostly on the default Cassandra > configuration at this time - only change is the max connection pool size > (CassandraHostConfigurator.maxActive) is changed to 300 from a default of > 50. > > I would also like to add that the time increase is not linear - it starts > fast, goes, slow, very slow, and becomes faster again. > > ------------------------ > 25% 29 > 50% 105 > 66% 185 > 70% 208 > 75% 240 > 80% 297 > 90% 510 > 95% 854 > 98% 1075 > 99% 1215 > 100% 4442 > ------------------------ > > -- > Roshan > Blog: http://roshandawrani.wordpress.com/ > Twitter: @roshandawrani > Skype: roshandawrani > > --20cf307d01c2e64545049e230fff Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Sounds like GC from your description of fast->slow-&= gt;fast. Collect GC times from both the client and server side and plot aga= inst your application timing.

=A0If you uncomment = the verbose GC entries in the cassandra-env.sh file you should get timing f= or the server side, pass in the same arguments for your client. Align time = across the 3 files and plot to see if GC is the cause.

=A0Sridhar



On Thu, Mar 10, 2011 at 9:30 AM, Roshan Dawrani <roshandawrani@gmail.com> wrote:
Hi,

I am in the middle o= f some load testing on a 1-node Cassandra setup. We are not on very high lo= ads yet. We have recorded the timings taken up by mutator.execute() calls a= nd we see this kind of variation during the test run:

So, 25% of the times, execute() calls come back in 25 m= illi-seconds, but the longer calls go upto 4 seconds.

<= div>Can someone please provide some pointers on what and where to focus on = in my Hector / Cassandra setup? We are mostly on the default Cassandra conf= iguration at this time - only change is the max connection pool size (Cassa= ndraHostConfigurator.maxActive) is changed to 300 from a default of 50.


--20cf307d01c2e64545049e230fff--