cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rohit bhatia <rohit2...@gmail.com>
Subject Re: Finding bottleneck of a cluster
Date Fri, 06 Jul 2012 04:14:33 GMT
On Fri, Jul 6, 2012 at 4:47 AM, aaron morton <aaron@thelastpickle.com> wrote:
> 12G Heap,
> 1600Mb Young gen,
>
> Is a bit higher than the normal recommendation. 1600MB young gen can cause
> some extra ParNew pauses.
Thanks for heads up, i'll try tinkering on this

>
> 128 Concurrent writer
> threads
>
> Unless you are on SSD this is too many.
>
I mean http://www.datastax.com/docs/0.8/configuration/node_configuration#concurrent-writes
, this is not memtable flush queue writers.
Suggested value is 8*number of cores(16) = 128 itself.
>
> 1) Is using JDK 1.7 any way detrimental to cassandra?
>
> as far as I know it's not fully certified, thanks for trying it :)
>
> 2) What is the max write operation qps that should be expected. Is the
> netflix benchmark also applicable for counter incrmenting tasks?
>
> Counters use a different write path than normal writes and are a bit slower.
>
> To benchmark, get a single node and work out the max throughput. Then
> multiply by the number of nodes and divide by the RF to get a rough idea.
>
> the cpu
> idle time is around 30%, cassandra is not disk bound(insignificant
> read operations and cpu's iowait is around 0.05%)
>
> Wait until compaction kicks in and handle all your inserts.
>
> The os load is around 16-20 and the average write latency is 3ms.
> tpstats do not show any significant pending tasks.
>
> The node is overloaded. What is the write latency for a single thread doing
> as single increment against a node that has not other traffic ? The latency
> for a request is the time spent working and the time spent waiting, once you
> read the max throughput the time spent waiting increases. The SEDA
> architecture is designed to limit the time spent working.
>
>    At this point suddenly, Several nodes start dropping several
> "Mutation" messages. There are also lots of pending
>
> The cluster is overwhelmed.
>
>  Almost all the new threads seem to be named
> "pool-2-thread-*".
>
> These are client connection threads.
>
> My guess is that this might be due to the 128 Writer threads not being
> able to perform more writes.(
>
> Yes.
> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L214
>
> Work out the latency for a single client single node, then start adding
> replication, nodes and load. When the latency increases you are getting to
> the max throughput for that config.

Also, as mentioned in my second mail, seeing messages like this "Total
time for which application threads were stopped: 16.7663710 seconds",
if something pauses for this long, it might be overwhelmed by the
hints stored at other nodes. This can further cause the node to wait
on/drop a lot of client connection threads. I'll look into what is
causing these non-gc pauses. Thanks for the help.

>
> Hope that helps
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 5/07/2012, at 6:49 PM, rohit bhatia wrote:
>
> Our Cassandra cluster consists of 8 nodes(16 core, 32G ram, 12G Heap,
> 1600Mb Young gen, cassandra1.0.5, JDK 1.7, 128 Concurrent writer
> threads). The replication factor is 2 with 10 column families and we
> service Counter incrementing write intensive tasks(CL=ONE).
>
> I am trying to figure out the bottleneck,
>
> 1) Is using JDK 1.7 any way detrimental to cassandra?
>
> 2) What is the max write operation qps that should be expected. Is the
> netflix benchmark also applicable for counter incrmenting tasks?
>
> http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html
>
> 3) At around 50,000qps for the cluster (~12500 qps per node), the cpu
> idle time is around 30%, cassandra is not disk bound(insignificant
> read operations and cpu's iowait is around 0.05%) and is not swapping
> its memory(around 15 gb RAM is free or inactive). The average gc pause
> time for parnew are 100ms occuring every second. So cassandra spends
> 10% of its time stuck in "Stop the world" collector.
> The os load is around 16-20 and the average write latency is 3ms.
> tpstats do not show any significant pending tasks.
>
>    At this point suddenly, Several nodes start dropping several
> "Mutation" messages. There are also lots of pending
> MutationStage,replicateOnWriteStage tasks in tpstats.
> The number of threads in the java process increase to around 25,000
> from the usual 300-400. Almost all the new threads seem to be named
> "pool-2-thread-*".
> The OS load jumps to around 30-40, the "write request latency" starts
> spiking to more than 500ms (even to several tens of seconds sometime).
> Even the "Local write latency" increases fourfolds to 200 microseconds
> from 50 microseconds. This happens across all the nodes and in around
> 2-3 minutes.
> My guess is that this might be due to the 128 Writer threads not being
> able to perform more writes.(though with  average local write latency
> of 100-150 micro seconds, each thread should be able to serve 10,000
> qps and with 128 writer threads, should be able to serve 1,280,000 qps
> per node)
> Could there be any other reason for this? What else should I monitor
> since system.log do not seem to say anything conclusive before
> dropping messages.
>
>
>
> Thanks
> Rohit
>
>

Mime
View raw message