cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: what causes MESSAGE-DESERIALIZER-POOL to spike
Date Tue, 27 Jul 2010 03:30:26 GMT
MDP is backing up because RRS is full at 4096.  This means you're not
able to process reads as quickly as the requests are coming in.  Make
whatever is doing those reads be less aggressive.

As to why the reads are slow in the first place, usually this means
you are disk i/o bound.  Posting your cfstats can help troubleshoot
but is no substitute for thinking about your application workload.

On Mon, Jul 26, 2010 at 12:32 PM, Anthony Molinaro
<anthonym@alumni.caltech.edu> wrote:
> It's usually I/O which causes backup in MESSAGE-DESERIALIZER-POOL.  You
> should check iostat and see what it looks like.  It may be that you
> need more nodes in order to deal with the read/write rate.   You can also
> use JMX to get latency values on reads and writes and see if the backup
> has a corresponding increase in latency.  You may be able to get more
> out of your hardware and memory with row caching but that really depends
> on your data set.
>
> -Anthony
>
> On Mon, Jul 26, 2010 at 12:22:46PM -0700, Dathan Pattishall wrote:
>> I have 4 nodes on enterprise type hardware (Lots of Ram 12GB, 16 i7 cores,
>> RAID Disks).
>>
>> ~# /opt/cassandra/bin/nodetool --host=localhost --port=8181 tpstats
>> Pool Name                    Active   Pending      Completed
>> STREAM-STAGE                      0         0              0
>> RESPONSE-STAGE                    0         0         516280
>> ROW-READ-STAGE                    8      4096        1164326
>> LB-OPERATIONS                     0         0              0
>> *MESSAGE-DESERIALIZER-POOL         1    682008        1818682*
>> GMFD                              0         0           6467
>> LB-TARGET                         0         0              0
>> CONSISTENCY-MANAGER               0         0         661477
>> ROW-MUTATION-STAGE                0         0         998780
>> MESSAGE-STREAMING-POOL            0         0              0
>> LOAD-BALANCER-STAGE               0         0              0
>> FLUSH-SORTER-POOL                 0         0              0
>> MEMTABLE-POST-FLUSHER             0         0              4
>> FLUSH-WRITER-POOL                 0         0              4
>> AE-SERVICE-STAGE                  0         0              0
>> HINTED-HANDOFF-POOL               0         0              3
>>
>> EQX root@cass04:~# vmstat -n 1
>>
>> procs -----------memory---------- ---swap-- -----io---- --system--
>> -----cpu------
>>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us
sy id
>> wa st
>>  6 10   7096 121816  16244 10375492    0    0     1     3    0    0
 5  1
>> 94  0  0
>>  2 10   7096 116484  16248 10381144    0    0  5636     4 21210 9820  2
 1
>> 79 18  0
>>  1  9   7096 108920  16248 10387592    0    0  6216     0 21439 9878  2
 1
>> 81 16  0
>>  0  9   7096 129108  16248 10364852    0    0  6024     0 23280 8753  2
 1
>> 80 17  0
>>  2  9   7096 122460  16248 10370908    0    0  6072     0 20835 9461  2
 1
>> 83 14  0
>>  2  8   7096 115740  16260 10375752    0    0  5168   292 21049 9511  3
 1
>> 77 20  0
>>  1 10   7096 108424  16260 10382300    0    0  6244     0 21483 8981  2
 1
>> 75 22  0
>>  3  8   7096 125028  16260 10364104    0    0  5584     0 21238 9436  2
 1
>> 81 16  0
>>  3  9   7096 117928  16260 10370064    0    0  5988     0 21505 10225
 2  1
>> 77 19  0
>>  1  8   7096 109544  16260 10376640    0    0  6340    28 20840 8602  2
 1
>> 80 18  0
>>  0  9   7096 127028  16240 10357652    0    0  5984     0 20853 9158  2
 1
>> 79 18  0
>>  9  0   7096 121472  16240 10363492    0    0  5716     0 20520 8489  1
 1
>> 82 16  0
>>  3  9   7096 112668  16240 10369872    0    0  6404     0 21314 9459  2
 1
>> 84 13  0
>>  1  9   7096 127300  16236 10353440    0    0  5684     0 38914 10068
 2  1
>> 76 21  0
>>
>>
>> *But the 16 cores are hardly utilized. Which indicates to me there is some
>> bad thread thrashing, but why? *
>>
>>
>>
>>   1  [|||||                                              
8.3%]     Tasks:
>> 1070 total, 1 running
>>   2  [                                                
   0.0%]     Load
>> average: 8.34 9.05 8.82
>>   3  [                                                
   0.0%]     Uptime:
>> 192 days(!), 15:29:52
>>   4  [|||||||||||                                        17.9%]
>>   5  [|||||                                              
5.7%]
>>   6  [||                                                
 1.3%]
>>   7  [||                                                
 2.6%]
>>   8  [|                                                
  0.6%]
>>   9  [|                                                
  0.6%]
>>   10 [||                                                
 1.9%]
>>   11 [||                                                
 1.9%]
>>   12 [||                                                
 1.9%]
>>   13 [||                                                
 1.3%]
>>   14 [|                                                
  0.6%]
>>   15 [||                                                
 1.3%]
>>   16 [|                                                
  0.6%]
>>   Mem[||||||||||||||||||||||||||||||||||||||||||||1791/12028MB]
>>   Swp[|                                               6/1983MB]
>>
>>   PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
>> 30269 root      40   0 14100  2116   900 R  4.0  0.0  0:00.49 htop
>> 24878 root      40   0 20.6G 8345M 6883M D  3.0 69.4  1:23.03
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24879 root      40   0 20.6G 8345M 6883M D  3.0 69.4  1:22.93
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24874 root      40   0 20.6G 8345M 6883M D  2.0 69.4  1:22.73
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24880 root      40   0 20.6G 8345M 6883M D  2.0 69.4  1:22.93
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24875 root      40   0 20.6G 8345M 6883M D  2.0 69.4  1:23.17
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24658 root      40   0 20.6G 8345M 6883M D  2.0 69.4  1:23.06
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24877 root      40   0 20.6G 8345M 6883M S  2.0 69.4  1:23.43
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24873 root      40   0 20.6G 8345M 6883M D  1.0 69.4  1:23.65
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24876 root      40   0 20.6G 8345M 6883M S  1.0 69.4  1:23.62
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24942 root      40   0 20.6G 8345M 6883M S  1.0 69.4  0:23.50
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24943 root      40   0 20.6G 8345M 6883M S  0.0 69.4  0:29.53
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24933 root      40   0 20.6G 8345M 6883M S  0.0 69.4  0:22.57
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 24939 root      40   0 20.6G 8345M 6883M S  0.0 69.4  0:12.73
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>> 25280 root      40   0 20.6G 8345M 6883M S  0.0 69.4  0:00.10
>> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym@alumni.caltech.edu>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message