Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 67A4CE21A for ; Tue, 27 Nov 2012 21:45:05 +0000 (UTC) Received: (qmail 61452 invoked by uid 500); 27 Nov 2012 21:45:02 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 61430 invoked by uid 500); 27 Nov 2012 21:45:02 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 61420 invoked by uid 99); 27 Nov 2012 21:45:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Nov 2012 21:45:02 +0000 X-ASF-Spam-Status: No, hits=2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com designates 209.85.223.172 as permitted sender) Received: from [209.85.223.172] (HELO mail-ie0-f172.google.com) (209.85.223.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Nov 2012 21:44:58 +0000 Received: by mail-ie0-f172.google.com with SMTP id c13so12627475ieb.31 for ; Tue, 27 Nov 2012 13:44:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=JBinpgEVzVszrAPRB27Tuw2FSiHDVoeIqm5qUsZp/Is=; b=jhNNsv5Y0RxuHgEldFHU54ZiuOb5y5KRMSTj+h2fXYyR4C1lYotaD/rfdElUo7m6Nr /zMUSdZZPqQ0LZJDD3IWEcUlZCQsEd5ZxFGBXSd8HAmdKyVbETDV3AICO4R+GZJ25dPb KFDb3TBhgR2jARC2ULAYbIXEkZLS3sI5JwYuR1c2MzPpxgmSPYZ+4ZIS8mh3Hr3afsLq ydAlmLSJtMUAWOhvNekDuF3A5M0mqtD5pSliAk5LkjaVakz70pOu4rDl+nKkhBRR7ftn 4Wr3scOj83bXbnBbK1Rvk3wBa0RVYfFCZz9eeDTPGfX9Rg+OHZWQz6n8bslZIguauJ9x /I6Q== MIME-Version: 1.0 Received: by 10.50.237.103 with SMTP id vb7mr17163369igc.29.1354052677728; Tue, 27 Nov 2012 13:44:37 -0800 (PST) Received: by 10.64.97.106 with HTTP; Tue, 27 Nov 2012 13:44:37 -0800 (PST) In-Reply-To: References: <1354038762579-7583996.post@n2.nabble.com> Date: Tue, 27 Nov 2012 16:44:37 -0500 Message-ID: Subject: Re: counters + replication = awful performance? From: Edward Capriolo To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=14dae9341205dcfb7004cf80f499 X-Virus-Checked: Checked by ClamAV on apache.org --14dae9341205dcfb7004cf80f499 Content-Type: text/plain; charset=ISO-8859-1 The difference between Replication factor =1 and replication factor > 1 is significant. Also it sounds like your cluster is 2 node so going from RF=1 to RF=2 means double the load on both nodes. You may want to experiment with the very dangerous column family attribute: - replicate_on_write: Replicate every counter update from the leader to the follower replicas. Accepts the values true and false. Edward On Tue, Nov 27, 2012 at 1:02 PM, Michael Kjellman wrote: > Are you writing with QUORUM consistency or ONE? > > On 11/27/12 9:52 AM, "Sergey Olefir" wrote: > > >Hi Juan, > > > >thanks for your input! > > > >In my case, however, I doubt this is the case -- clients are able to push > >many more updates than I need to saturate replication_factor=2 case (e.g. > >I'm doing as many as 6x more increments when testing 2-node cluster with > >replication_factor=1), so bandwidth between clients and server should be > >sufficient. > > > >Bandwidth between nodes in the cluster should also be quite sufficient > >since > >they are both in the same DC. But it is something to check, thanks! > > > >Best regards, > >Sergey > > > > > >Juan Valencia wrote > >> Hi Sergey, > >> > >> I know I've had similar issues with counters which were bottle-necked by > >> network throughput. You might be seeing a problem with throughput > >>between > >> the clients and Cass or between the two Cass nodes. It might not be > >>your > >> case, but that was what happened to me :-) > >> > >> Juan > >> > >> > >> On Tue, Nov 27, 2012 at 8:48 AM, Sergey Olefir < > > > >> solf.lists@ > > > >> > wrote: > >> > >>> Hi, > >>> > >>> I have a serious problem with counters performance and I can't seem to > >>> figure it out. > >>> > >>> Basically I'm building a system for accumulating some statistics "on > >>>the > >>> fly" via Cassandra distributed counters. For this I need counter > >>>updates > >>> to > >>> work "really fast" and herein lies my problem -- as soon as I enable > >>> replication_factor = 2, the performance goes down the drain. This > >>>happens > >>> in > >>> my tests using both 1.0.x and 1.1.6. > >>> > >>> Let me elaborate: > >>> > >>> I have two boxes (virtual servers on top of physical servers rented > >>> specifically for this purpose, i.e. it's not a cloud, nor it is shared; > >>> virtual servers are managed by our admins as a way to limit damage as I > >>> suppose :)). Cassandra partitioner is set to ByteOrderedPartitioner > >>> because > >>> I want to be able to do some range queries. > >>> > >>> First, I set up Cassandra individually on each box (not in a cluster) > >>>and > >>> test counter increments performance (exclusively increments, no reads). > >>> For > >>> tests I use code that is intended to somewhat resemble the expected > >>>load > >>> pattern -- particularly the majority of increments create new counters > >>> with > >>> some updating (adding) to already existing counters. In this test each > >>> single node exhibits respectable performance - something on the order > >>>of > >>> 70k > >>> (seventy thousand) increments per second. > >>> > >>> I then join both of these nodes into single cluster (using SimpleSnitch > >>> and > >>> SimpleStrategy, nothing fancy yet). I then run the same test using > >>> replication_factor=1. The performance is on the order of 120k > >>>increments > >>> per > >>> second -- which seems to be a reasonable increase over the single node > >>> performance. > >>> > >>> > >>> HOWEVER I then rerun the same test on the two-node cluster using > >>> replication_factor=2 -- which is the least I'll need for actual > >>> production > >>> for redundancy purposes. And the performance I get is absolutely > >>>horrible > >>> -- > >>> much, MUCH worse than even single-node performance -- something on the > >>> order > >>> of less than 25k increments per second. In addition to clients not > >>>being > >>> able to push updates fast enough, I also see a lot of 'messages > >>>dropped' > >>> messages in the Cassandra log under this load. > >>> > >>> Could anyone advise what could be causing such drastic performance drop > >>> under replication_factor=2? I was expecting something on the order of > >>> single-node performance, not approximately 3x less. > >>> > >>> > >>> When testing replication_factor=2 on 1.1.6 I can see that CPU usage > >>>goes > >>> through the roof. On 1.0.x I think it looked more like disk overload, > >>>but > >>> I'm not sure (being on virtual server I apparently can't see true > >>> iostats). > >>> > >>> I do have Cassandra data on a separate disk, commit log and cache are > >>> currently on the same disk as the system. I experimented with commit > >>>log > >>> flush modes and even with disabling commit log at all -- but it doesn't > >>> seem > >>> to have noticeable impact on the performance when under > >>> replication_factor=2. > >>> > >>> > >>> Any suggestions and hints will be much appreciated :) And please let me > >>> know > >>> if I need to share additional information about the configuration I'm > >>> running on. > >>> > >>> Best regards, > >>> Sergey > >>> > >>> > >>> > >>> -- > >>> View this message in context: > >>> > >>> > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/counter > >>>s-replication-awful-performance-tp7583993.html > >>> Sent from the > > > >> cassandra-user@.apache > > > >> mailing list archive at > >>> Nabble.com. > >>> > >> > >> > >> > >> -- > >> > >> Learn More: SQI (Social Quality Index) - A Universal Measure of Social > >> Quality > > > > > > > > > > > >-- > >View this message in context: > > > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/counters- > >replication-awful-performance-tp7583993p7583996.html > >Sent from the cassandra-user@incubator.apache.org mailing list archive at > >Nabble.com. > > > 'Like' us on Facebook for exclusive content and other resources on all > Barracuda Networks solutions. > > Visit http://barracudanetworks.com/facebook > > > > > --14dae9341205dcfb7004cf80f499 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
The difference between Replication factor =3D1 and replication factor = > 1 is significant. Also it sounds like your cluster is 2 node so going = from RF=3D1 to RF=3D2 means double the load on both nodes.
You may want to experiment with the very dangerous column family attri= bute:

- replicate_on_write: Replicate every counte= r update from the leader to the
follower replicas. Accepts the values tr= ue and false.

Edward
On Tue= , Nov 27, 2012 at 1:02 PM, Michael Kjellman <mkjellman@barracuda.c= om> wrote:
Are you writing with QUORUM consistency or O= NE?

On 11/27/12 9:52 AM, "Sergey Olefir" <solf.lists@gmail.com> wrote:

>Hi Juan,
>
>thanks for your input!
>
>In my case, however, I doubt this is the case -- clients are able to pu= sh
>many more updates than I need to saturate replication_factor=3D2 case (= e.g.
>I'm doing as many as 6x more increments when testing 2-node cluster= with
>replication_factor=3D1), so bandwidth between clients and server should= be
>sufficient.
>
>Bandwidth between nodes in the cluster should also be quite sufficient<= br> >since
>they are both in the same DC. But it is something to check, thanks!
>
>Best regards,
>Sergey
>
>
>Juan Valencia wrote
>> Hi Sergey,
>>
>> I know I've had similar issues with counters which were bottle= -necked by
>> network throughput. =A0You might be seeing a problem with throughp= ut
>>between
>> the clients and Cass or between the two Cass nodes. =A0It might no= t be
>>your
>> case, but that was what happened to me :-)
>>
>> Juan
>>
>>
>> On Tue, Nov 27, 2012 at 8:48 AM, Sergey Olefir &lt;
>
>> solf.lists@
>
>> &gt; wrote:
>>
>>> Hi,
>>>
>>> I have a serious problem with counters performance and I can&#= 39;t seem to
>>> figure it out.
>>>
>>> Basically I'm building a system for accumulating some stat= istics "on
>>>the
>>> fly" via Cassandra distributed counters. For this I need = counter
>>>updates
>>> to
>>> work "really fast" and herein lies my problem -- as = soon as I enable
>>> replication_factor =3D 2, the performance goes down the drain.= This
>>>happens
>>> in
>>> my tests using both 1.0.x and 1.1.6.
>>>
>>> Let me elaborate:
>>>
>>> I have two boxes (virtual servers on top of physical servers r= ented
>>> specifically for this purpose, i.e. it's not a cloud, nor = it is shared;
>>> virtual servers are managed by our admins as a way to limit da= mage as I
>>> suppose :)). Cassandra partitioner is set to ByteOrderedPartit= ioner
>>> because
>>> I want to be able to do some range queries.
>>>
>>> First, I set up Cassandra individually on each box (not in a c= luster)
>>>and
>>> test counter increments performance (exclusively increments, n= o reads).
>>> For
>>> tests I use code that is intended to somewhat resemble the exp= ected
>>>load
>>> pattern -- particularly the majority of increments create new = counters
>>> with
>>> some updating (adding) to already existing counters. In this t= est each
>>> single node exhibits respectable performance - something on th= e order
>>>of
>>> 70k
>>> (seventy thousand) increments per second.
>>>
>>> I then join both of these nodes into single cluster (using Sim= pleSnitch
>>> and
>>> SimpleStrategy, nothing fancy yet). I then run the same test u= sing
>>> replication_factor=3D1. The performance is on the order of 120= k
>>>increments
>>> per
>>> second -- which seems to be a reasonable increase over the sin= gle node
>>> performance.
>>>
>>>
>>> HOWEVER I then rerun the same test on the two-node cluster usi= ng
>>> replication_factor=3D2 -- which is the least I'll need for= actual
>>> production
>>> for redundancy purposes. And the performance I get is absolute= ly
>>>horrible
>>> --
>>> much, MUCH worse than even single-node performance -- somethin= g on the
>>> order
>>> of less than 25k increments per second. In addition to clients= not
>>>being
>>> able to push updates fast enough, I also see a lot of 'mes= sages
>>>dropped'
>>> messages in the Cassandra log under this load.
>>>
>>> Could anyone advise what could be causing such drastic perform= ance drop
>>> under replication_factor=3D2? I was expecting something on the= order of
>>> single-node performance, not approximately 3x less.
>>>
>>>
>>> When testing replication_factor=3D2 on 1.1.6 I can see that CP= U usage
>>>goes
>>> through the roof. On 1.0.x I think it looked more like disk ov= erload,
>>>but
>>> I'm not sure (being on virtual server I apparently can'= ;t see true
>>> iostats).
>>>
>>> I do have Cassandra data on a separate disk, commit log and ca= che are
>>> currently on the same disk as the system. I experimented with = commit
>>>log
>>> flush modes and even with disabling commit log at all -- but i= t doesn't
>>> seem
>>> to have noticeable impact on the performance when under
>>> replication_factor=3D2.
>>>
>>>
>>> Any suggestions and hints will be much appreciated :) And plea= se let me
>>> know
>>> if I need to share additional information about the configurat= ion I'm
>>> running on.
>>>
>>> Best regards,
>>> Sergey
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>>
>>>http://cassandra-user-incubator-apa= che-org.3065146.n2.nabble.com/counter
>>>s-replication-awful-performance-tp7583993.html
>>> Sent from the
>
>> cassandra-user@.apache
>
>> =A0mailing list archive at
>>> Nabble.com.
>>>
>>
>>
>>
>> --
>>
>> Learn More: =A0SQI (Social Quality Index) - A Universal Measure of= Social
>> Quality
>
>
>
>
>
>--
>View this message in context:
>http://cassandra-user-incubator-apache-or= g.3065146.n2.nabble.com/counters-
>replication-awful-performance-tp7583993p7583996.html
>Sent from the ca= ssandra-user@incubator.apache.org mailing list archive at
>Nabble.com.


'Like' us on Facebook for exclusive content and other resources on = all Barracuda Networks solutions.

Visit h= ttp://barracudanetworks.com/facebook





--14dae9341205dcfb7004cf80f499--