incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott McKay <sco...@mailchannels.com>
Subject Re: counters + replication = awful performance?
Date Tue, 27 Nov 2012 23:13:32 GMT
We're having a similar performance problem.  Setting
'replicate_on_write: false' fixes the performance issue in our tests.

How dangerous is it?  What exactly could go wrong?

On 12-11-27 01:44 PM, Edward Capriolo wrote:
> The difference between Replication factor =1 and replication factor >
> 1 is significant. Also it sounds like your cluster is 2 node so going
> from RF=1 to RF=2 means double the load on both nodes.
>
> You may want to experiment with the very dangerous column family
> attribute:
>
> - replicate_on_write: Replicate every counter update from the leader
> to the
> follower replicas. Accepts the values true and false.
>
> Edward
> On Tue, Nov 27, 2012 at 1:02 PM, Michael Kjellman
> <mkjellman@barracuda.com <mailto:mkjellman@barracuda.com>> wrote:
>
>     Are you writing with QUORUM consistency or ONE?
>
>     On 11/27/12 9:52 AM, "Sergey Olefir" <solf.lists@gmail.com
>     <mailto:solf.lists@gmail.com>> wrote:
>
>     >Hi Juan,
>     >
>     >thanks for your input!
>     >
>     >In my case, however, I doubt this is the case -- clients are able
>     to push
>     >many more updates than I need to saturate replication_factor=2
>     case (e.g.
>     >I'm doing as many as 6x more increments when testing 2-node
>     cluster with
>     >replication_factor=1), so bandwidth between clients and server
>     should be
>     >sufficient.
>     >
>     >Bandwidth between nodes in the cluster should also be quite
>     sufficient
>     >since
>     >they are both in the same DC. But it is something to check, thanks!
>     >
>     >Best regards,
>     >Sergey
>     >
>     >
>     >Juan Valencia wrote
>     >> Hi Sergey,
>     >>
>     >> I know I've had similar issues with counters which were
>     bottle-necked by
>     >> network throughput.  You might be seeing a problem with throughput
>     >>between
>     >> the clients and Cass or between the two Cass nodes.  It might
>     not be
>     >>your
>     >> case, but that was what happened to me :-)
>     >>
>     >> Juan
>     >>
>     >>
>     >> On Tue, Nov 27, 2012 at 8:48 AM, Sergey Olefir &lt;
>     >
>     >> solf.lists@
>     >
>     >> &gt; wrote:
>     >>
>     >>> Hi,
>     >>>
>     >>> I have a serious problem with counters performance and I can't
>     seem to
>     >>> figure it out.
>     >>>
>     >>> Basically I'm building a system for accumulating some
>     statistics "on
>     >>>the
>     >>> fly" via Cassandra distributed counters. For this I need counter
>     >>>updates
>     >>> to
>     >>> work "really fast" and herein lies my problem -- as soon as I
>     enable
>     >>> replication_factor = 2, the performance goes down the drain. This
>     >>>happens
>     >>> in
>     >>> my tests using both 1.0.x and 1.1.6.
>     >>>
>     >>> Let me elaborate:
>     >>>
>     >>> I have two boxes (virtual servers on top of physical servers
>     rented
>     >>> specifically for this purpose, i.e. it's not a cloud, nor it
>     is shared;
>     >>> virtual servers are managed by our admins as a way to limit
>     damage as I
>     >>> suppose :)). Cassandra partitioner is set to
>     ByteOrderedPartitioner
>     >>> because
>     >>> I want to be able to do some range queries.
>     >>>
>     >>> First, I set up Cassandra individually on each box (not in a
>     cluster)
>     >>>and
>     >>> test counter increments performance (exclusively increments,
>     no reads).
>     >>> For
>     >>> tests I use code that is intended to somewhat resemble the
>     expected
>     >>>load
>     >>> pattern -- particularly the majority of increments create new
>     counters
>     >>> with
>     >>> some updating (adding) to already existing counters. In this
>     test each
>     >>> single node exhibits respectable performance - something on
>     the order
>     >>>of
>     >>> 70k
>     >>> (seventy thousand) increments per second.
>     >>>
>     >>> I then join both of these nodes into single cluster (using
>     SimpleSnitch
>     >>> and
>     >>> SimpleStrategy, nothing fancy yet). I then run the same test using
>     >>> replication_factor=1. The performance is on the order of 120k
>     >>>increments
>     >>> per
>     >>> second -- which seems to be a reasonable increase over the
>     single node
>     >>> performance.
>     >>>
>     >>>
>     >>> HOWEVER I then rerun the same test on the two-node cluster using
>     >>> replication_factor=2 -- which is the least I'll need for actual
>     >>> production
>     >>> for redundancy purposes. And the performance I get is absolutely
>     >>>horrible
>     >>> --
>     >>> much, MUCH worse than even single-node performance --
>     something on the
>     >>> order
>     >>> of less than 25k increments per second. In addition to clients not
>     >>>being
>     >>> able to push updates fast enough, I also see a lot of 'messages
>     >>>dropped'
>     >>> messages in the Cassandra log under this load.
>     >>>
>     >>> Could anyone advise what could be causing such drastic
>     performance drop
>     >>> under replication_factor=2? I was expecting something on the
>     order of
>     >>> single-node performance, not approximately 3x less.
>     >>>
>     >>>
>     >>> When testing replication_factor=2 on 1.1.6 I can see that CPU
>     usage
>     >>>goes
>     >>> through the roof. On 1.0.x I think it looked more like disk
>     overload,
>     >>>but
>     >>> I'm not sure (being on virtual server I apparently can't see true
>     >>> iostats).
>     >>>
>     >>> I do have Cassandra data on a separate disk, commit log and
>     cache are
>     >>> currently on the same disk as the system. I experimented with
>     commit
>     >>>log
>     >>> flush modes and even with disabling commit log at all -- but
>     it doesn't
>     >>> seem
>     >>> to have noticeable impact on the performance when under
>     >>> replication_factor=2.
>     >>>
>     >>>
>     >>> Any suggestions and hints will be much appreciated :) And
>     please let me
>     >>> know
>     >>> if I need to share additional information about the
>     configuration I'm
>     >>> running on.
>     >>>
>     >>> Best regards,
>     >>> Sergey
>     >>>
>     >>>
>     >>>
>     >>> --
>     >>> View this message in context:
>     >>>
>     >>>http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/counter
>     >>>s-replication-awful-performance-tp7583993.html
>     >>> Sent from the
>     >
>     >> cassandra-user@.apache
>     >
>     >>  mailing list archive at
>     >>> Nabble.com.
>     >>>
>     >>
>     >>
>     >>
>     >> --
>     >>
>     >> Learn More:  SQI (Social Quality Index) - A Universal Measure
>     of Social
>     >> Quality
>     >
>     >
>     >
>     >
>     >
>     >--
>     >View this message in context:
>     >http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/counters-
>     >replication-awful-performance-tp7583993p7583996.html
>     >Sent from the cassandra-user@incubator.apache.org
>     <mailto:cassandra-user@incubator.apache.org> mailing list archive at
>     >Nabble.com.
>
>
>     'Like' us on Facebook for exclusive content and other resources on
>     all Barracuda Networks solutions.
>
>     Visit http://barracudanetworks.com/facebook
>
>
>
>
>

-- 
*Scott McKay*, Sr. Software Developer
MailChannels

Tel: +1 604 685 7488 x 509
www.mailchannels.com

Mime
View raw message