incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Black...@b3k.us>
Subject Re: what causes MESSAGE-DESERIALIZER-POOL to spike
Date Sat, 31 Jul 2010 22:45:40 GMT
Sorry, I just noticed: are you running 14 instances of Cassandra on a
single physical machine or are all those java processes something
else?

On Mon, Jul 26, 2010 at 12:22 PM, Dathan Pattishall <dathanvp@gmail.com> wrote:
> I have 4 nodes on enterprise type hardware (Lots of Ram 12GB, 16 i7 cores,
> RAID Disks).
>
> ~# /opt/cassandra/bin/nodetool --host=localhost --port=8181 tpstats
> Pool Name                    Active   Pending      Completed
> STREAM-STAGE                      0         0             
0
> RESPONSE-STAGE                    0         0        
516280
> ROW-READ-STAGE                    8      4096        1164326
> LB-OPERATIONS                     0         0             
0
> MESSAGE-DESERIALIZER-POOL         1    682008        1818682
> GMFD                              0         0          
6467
> LB-TARGET                         0         0             
0
> CONSISTENCY-MANAGER               0         0         661477
> ROW-MUTATION-STAGE                0         0        
998780
> MESSAGE-STREAMING-POOL            0         0             
0
> LOAD-BALANCER-STAGE               0         0             
0
> FLUSH-SORTER-POOL                 0         0             
0
> MEMTABLE-POST-FLUSHER             0         0             
4
> FLUSH-WRITER-POOL                 0         0             
4
> AE-SERVICE-STAGE                  0         0             
0
> HINTED-HANDOFF-POOL               0         0             
3
>
> EQX root@cass04:~# vmstat -n 1
>
> procs -----------memory---------- ---swap-- -----io---- --system--
> -----cpu------
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in  
cs us sy id
> wa st
>  6 10   7096 121816  16244 10375492    0    0     1     3    0   
0  5  1
> 94  0  0
>  2 10   7096 116484  16248 10381144    0    0  5636     4 21210 9820 
2  1
> 79 18  0
>  1  9   7096 108920  16248 10387592    0    0  6216     0 21439 9878 
2  1
> 81 16  0
>  0  9   7096 129108  16248 10364852    0    0  6024     0 23280 8753 
2  1
> 80 17  0
>  2  9   7096 122460  16248 10370908    0    0  6072     0 20835 9461 
2  1
> 83 14  0
>  2  8   7096 115740  16260 10375752    0    0  5168   292 21049 9511 
3  1
> 77 20  0
>  1 10   7096 108424  16260 10382300    0    0  6244     0 21483 8981 
2  1
> 75 22  0
>  3  8   7096 125028  16260 10364104    0    0  5584     0 21238 9436 
2  1
> 81 16  0
>  3  9   7096 117928  16260 10370064    0    0  5988     0 21505 10225 
2  1
> 77 19  0
>  1  8   7096 109544  16260 10376640    0    0  6340    28 20840 8602 
2  1
> 80 18  0
>  0  9   7096 127028  16240 10357652    0    0  5984     0 20853 9158 
2  1
> 79 18  0
>  9  0   7096 121472  16240 10363492    0    0  5716     0 20520 8489 
1  1
> 82 16  0
>  3  9   7096 112668  16240 10369872    0    0  6404     0 21314 9459 
2  1
> 84 13  0
>  1  9   7096 127300  16236 10353440    0    0  5684     0 38914 10068 
2  1
> 76 21  0
>
>
> But the 16 cores are hardly utilized. Which indicates to me there is some
> bad thread thrashing, but why?
>
>
>
>   1  [|||||                                              
8.3%]     Tasks:
> 1070 total, 1 running
>   2  [                                                   
0.0%]     Load
> average: 8.34 9.05 8.82
>   3  [                                                   
0.0%]     Uptime:
> 192 days(!), 15:29:52
>   4  [|||||||||||                                       
17.9%]
>   5  [|||||                                              
5.7%]
>   6  [||                                                 
1.3%]
>   7  [||                                                 
2.6%]
>   8  [|                                                  
0.6%]
>   9  [|                                                  
0.6%]
>   10 [||                                                 
1.9%]
>   11 [||                                                 
1.9%]
>   12 [||                                                 
1.9%]
>   13 [||                                                 
1.3%]
>   14 [|                                                  
0.6%]
>   15 [||                                                 
1.3%]
>   16 [|                                                  
0.6%]
>   Mem[||||||||||||||||||||||||||||||||||||||||||||1791/12028MB]
>   Swp[|                                              
6/1983MB]
>
>   PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> 30269 root      40   0 14100  2116   900 R  4.0  0.0  0:00.49 htop
> 24878 root      40   0 20.6G 8345M 6883M D  3.0 69.4  1:23.03
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24879 root      40   0 20.6G 8345M 6883M D  3.0 69.4  1:22.93
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24874 root      40   0 20.6G 8345M 6883M D  2.0 69.4  1:22.73
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24880 root      40   0 20.6G 8345M 6883M D  2.0 69.4  1:22.93
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24875 root      40   0 20.6G 8345M 6883M D  2.0 69.4  1:23.17
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24658 root      40   0 20.6G 8345M 6883M D  2.0 69.4  1:23.06
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24877 root      40   0 20.6G 8345M 6883M S  2.0 69.4  1:23.43
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24873 root      40   0 20.6G 8345M 6883M D  1.0 69.4  1:23.65
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24876 root      40   0 20.6G 8345M 6883M S  1.0 69.4  1:23.62
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24942 root      40   0 20.6G 8345M 6883M S  1.0 69.4  0:23.50
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24943 root      40   0 20.6G 8345M 6883M S  0.0 69.4  0:29.53
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24933 root      40   0 20.6G 8345M 6883M S  0.0 69.4  0:22.57
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 24939 root      40   0 20.6G 8345M 6883M S  0.0 69.4  0:12.73
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
> 25280 root      40   0 20.6G 8345M 6883M S  0.0 69.4  0:00.10
> /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark
>
Mime
View raw message