cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rohit bhatia <rohit2...@gmail.com>
Subject Re: Finding bottleneck of a cluster
Date Fri, 06 Jul 2012 04:21:30 GMT
On Fri, Jul 6, 2012 at 9:44 AM, rohit bhatia <rohit2412@gmail.com> wrote:
> On Fri, Jul 6, 2012 at 4:47 AM, aaron morton <aaron@thelastpickle.com> wrote:
>> 12G Heap,
>> 1600Mb Young gen,
>>
>> Is a bit higher than the normal recommendation. 1600MB young gen can cause
>> some extra ParNew pauses.
> Thanks for heads up, i'll try tinkering on this
>
>>
>> 128 Concurrent writer
>> threads
>>
>> Unless you are on SSD this is too many.
>>
> I mean http://www.datastax.com/docs/0.8/configuration/node_configuration#concurrent-writes
> , this is not memtable flush queue writers.
> Suggested value is 8*number of cores(16) = 128 itself.
>>
>> 1) Is using JDK 1.7 any way detrimental to cassandra?
>>
>> as far as I know it's not fully certified, thanks for trying it :)
>>
>> 2) What is the max write operation qps that should be expected. Is the
>> netflix benchmark also applicable for counter incrmenting tasks?
>>
>> Counters use a different write path than normal writes and are a bit slower.
>>
>> To benchmark, get a single node and work out the max throughput. Then
>> multiply by the number of nodes and divide by the RF to get a rough idea.
>>
>> the cpu
>> idle time is around 30%, cassandra is not disk bound(insignificant
>> read operations and cpu's iowait is around 0.05%)
>>
>> Wait until compaction kicks in and handle all your inserts.
>>
>> The os load is around 16-20 and the average write latency is 3ms.
>> tpstats do not show any significant pending tasks.
>>
>> The node is overloaded. What is the write latency for a single thread doing
>> as single increment against a node that has not other traffic ? The latency
>> for a request is the time spent working and the time spent waiting, once you
>> read the max throughput the time spent waiting increases. The SEDA
>> architecture is designed to limit the time spent working.
The write latency I reported is as reported by datastax opscenter for
the total latency of a client's request. This is minimum at .5ms.
In contrast, the "local write request latency" as reported by cfstats
are around 50 micro seconds but jump to 150 microseconds during the
crash.


>>
>>    At this point suddenly, Several nodes start dropping several
>> "Mutation" messages. There are also lots of pending
>>
>> The cluster is overwhelmed.
>>
>>  Almost all the new threads seem to be named
>> "pool-2-thread-*".
>>
>> These are client connection threads.
>>
>> My guess is that this might be due to the 128 Writer threads not being
>> able to perform more writes.(
>>
>> Yes.
>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L214
>>
>> Work out the latency for a single client single node, then start adding
>> replication, nodes and load. When the latency increases you are getting to
>> the max throughput for that config.
>
> Also, as mentioned in my second mail, seeing messages like this "Total
> time for which application threads were stopped: 16.7663710 seconds",
> if something pauses for this long, it might be overwhelmed by the
> hints stored at other nodes. This can further cause the node to wait
> on/drop a lot of client connection threads. I'll look into what is
> causing these non-gc pauses. Thanks for the help.
>
>>
>> Hope that helps
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 5/07/2012, at 6:49 PM, rohit bhatia wrote:
>>
>> Our Cassandra cluster consists of 8 nodes(16 core, 32G ram, 12G Heap,
>> 1600Mb Young gen, cassandra1.0.5, JDK 1.7, 128 Concurrent writer
>> threads). The replication factor is 2 with 10 column families and we
>> service Counter incrementing write intensive tasks(CL=ONE).
>>
>> I am trying to figure out the bottleneck,
>>
>> 1) Is using JDK 1.7 any way detrimental to cassandra?
>>
>> 2) What is the max write operation qps that should be expected. Is the
>> netflix benchmark also applicable for counter incrmenting tasks?
>>
>> http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html
>>
>> 3) At around 50,000qps for the cluster (~12500 qps per node), the cpu
>> idle time is around 30%, cassandra is not disk bound(insignificant
>> read operations and cpu's iowait is around 0.05%) and is not swapping
>> its memory(around 15 gb RAM is free or inactive). The average gc pause
>> time for parnew are 100ms occuring every second. So cassandra spends
>> 10% of its time stuck in "Stop the world" collector.
>> The os load is around 16-20 and the average write latency is 3ms.
>> tpstats do not show any significant pending tasks.
>>
>>    At this point suddenly, Several nodes start dropping several
>> "Mutation" messages. There are also lots of pending
>> MutationStage,replicateOnWriteStage tasks in tpstats.
>> The number of threads in the java process increase to around 25,000
>> from the usual 300-400. Almost all the new threads seem to be named
>> "pool-2-thread-*".
>> The OS load jumps to around 30-40, the "write request latency" starts
>> spiking to more than 500ms (even to several tens of seconds sometime).
>> Even the "Local write latency" increases fourfolds to 200 microseconds
>> from 50 microseconds. This happens across all the nodes and in around
>> 2-3 minutes.
>> My guess is that this might be due to the 128 Writer threads not being
>> able to perform more writes.(though with  average local write latency
>> of 100-150 micro seconds, each thread should be able to serve 10,000
>> qps and with 128 writer threads, should be able to serve 1,280,000 qps
>> per node)
>> Could there be any other reason for this? What else should I monitor
>> since system.log do not seem to say anything conclusive before
>> dropping messages.
>>
>>
>>
>> Thanks
>> Rohit
>>
>>

Mime
View raw message