incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Valencia <jvalen...@sharethis.com>
Subject Re: counters + replication = awful performance?
Date Tue, 27 Nov 2012 17:38:51 GMT
Hi Sergey,

I know I've had similar issues with counters which were bottle-necked by
network throughput.  You might be seeing a problem with throughput between
the clients and Cass or between the two Cass nodes.  It might not be your
case, but that was what happened to me :-)

Juan


On Tue, Nov 27, 2012 at 8:48 AM, Sergey Olefir <solf.lists@gmail.com> wrote:

> Hi,
>
> I have a serious problem with counters performance and I can't seem to
> figure it out.
>
> Basically I'm building a system for accumulating some statistics "on the
> fly" via Cassandra distributed counters. For this I need counter updates to
> work "really fast" and herein lies my problem -- as soon as I enable
> replication_factor = 2, the performance goes down the drain. This happens
> in
> my tests using both 1.0.x and 1.1.6.
>
> Let me elaborate:
>
> I have two boxes (virtual servers on top of physical servers rented
> specifically for this purpose, i.e. it's not a cloud, nor it is shared;
> virtual servers are managed by our admins as a way to limit damage as I
> suppose :)). Cassandra partitioner is set to ByteOrderedPartitioner because
> I want to be able to do some range queries.
>
> First, I set up Cassandra individually on each box (not in a cluster) and
> test counter increments performance (exclusively increments, no reads). For
> tests I use code that is intended to somewhat resemble the expected load
> pattern -- particularly the majority of increments create new counters with
> some updating (adding) to already existing counters. In this test each
> single node exhibits respectable performance - something on the order of
> 70k
> (seventy thousand) increments per second.
>
> I then join both of these nodes into single cluster (using SimpleSnitch and
> SimpleStrategy, nothing fancy yet). I then run the same test using
> replication_factor=1. The performance is on the order of 120k increments
> per
> second -- which seems to be a reasonable increase over the single node
> performance.
>
>
> HOWEVER I then rerun the same test on the two-node cluster using
> replication_factor=2 -- which is the least I'll need for actual production
> for redundancy purposes. And the performance I get is absolutely horrible
> --
> much, MUCH worse than even single-node performance -- something on the
> order
> of less than 25k increments per second. In addition to clients not being
> able to push updates fast enough, I also see a lot of 'messages dropped'
> messages in the Cassandra log under this load.
>
> Could anyone advise what could be causing such drastic performance drop
> under replication_factor=2? I was expecting something on the order of
> single-node performance, not approximately 3x less.
>
>
> When testing replication_factor=2 on 1.1.6 I can see that CPU usage goes
> through the roof. On 1.0.x I think it looked more like disk overload, but
> I'm not sure (being on virtual server I apparently can't see true iostats).
>
> I do have Cassandra data on a separate disk, commit log and cache are
> currently on the same disk as the system. I experimented with commit log
> flush modes and even with disabling commit log at all -- but it doesn't
> seem
> to have noticeable impact on the performance when under
> replication_factor=2.
>
>
> Any suggestions and hints will be much appreciated :) And please let me
> know
> if I need to share additional information about the configuration I'm
> running on.
>
> Best regards,
> Sergey
>
>
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/counters-replication-awful-performance-tp7583993.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>



-- 

Learn More:  SQI (Social Quality Index) - A Universal Measure of Social
Quality

Mime
View raw message