cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan <cne...@yahoo.com>
Subject Re: cassandra node jvm stall intermittently
Date Wed, 04 Mar 2015 20:26:48 GMT
HI Jason; 
Whats in the log files at the moment jstat shows 100%. What is the activity on the cluster
& the node at the specific point in time (reads/ writes/ joins etc)
Jan/ 

     On Wednesday, March 4, 2015 5:59 AM, Jason Wee <peichieh@gmail.com> wrote:
   

 Hi, our cassandra node using java 7 update 72 and we ran jstat on one of the node, and notice
some strange behaviour as indicated by output below. any idea why when eden space stay the
same for few seconds like 100% and 18.02% for few seconds? we suspect such "stalling" cause
timeout to our cluster.
any idea what happened, what went wrong and what could cause this?

$ jstat -gcutil 32276 1s
  0.00   5.78  91.21  70.94  60.07   2657   73.437     4    0.056   73.493  0.00
  5.78 100.00  70.94  60.07   2657   73.437     4    0.056   73.493  0.00   5.78
100.00  70.94  60.07   2657   73.437     4    0.056   73.493  0.00   5.78 100.00
 70.94  60.07   2657   73.437     4    0.056   73.493  0.00   5.78 100.00  70.94
 60.07   2657   73.437     4    0.056   73.493  0.00   5.78 100.00  70.94  60.07
  2657   73.437     4    0.056   73.493  0.00   5.78 100.00  70.94  60.07   2657
  73.437     4    0.056   73.493  0.00   5.78 100.00  70.94  60.07   2657   73.437
    4    0.056   73.493  0.00   5.78 100.00  70.94  60.07   2657   73.437    
4    0.056   73.493  0.00   5.78 100.00  70.94  60.07   2657   73.437     4  
 0.056   73.493  0.00   5.78 100.00  70.94  60.07   2657   73.437     4    0.056
  73.493  0.00   5.78 100.00  70.94  60.07   2657   73.437     4    0.056   73.493 
0.00   5.78 100.00  70.94  60.07   2657   73.437     4    0.056   73.493  0.00
  5.78 100.00  70.94  60.07   2657   73.437     4    0.056   73.493  0.00   5.78
100.00  70.94  60.07   2657   73.437     4    0.056   73.493  0.00   5.78 100.00
 70.94  60.07   2657   73.437     4    0.056   73.493  0.00   5.78 100.00  70.94
 60.07   2657   73.437     4    0.056   73.493  0.00   4.65  29.66  71.00  60.07
  2659   73.488     4    0.056   73.544  0.00   4.65  70.88  71.00  60.07   2659
  73.488     4    0.056   73.544  0.00   4.65  71.58  71.00  60.07   2659   73.488
    4    0.056   73.544  0.00   4.65  72.15  71.00  60.07   2659   73.488    
4    0.056   73.544  0.00   4.65  72.33  71.00  60.07   2659   73.488     4  
 0.056   73.544  0.00   4.65  72.73  71.00  60.07   2659   73.488     4    0.056
  73.544  0.00   4.65  73.20  71.00  60.07   2659   73.488     4    0.056   73.544 
0.00   4.65  73.71  71.00  60.07   2659   73.488     4    0.056   73.544  0.00
  4.65  73.84  71.00  60.07   2659   73.488     4    0.056   73.544  0.00   4.65
 73.91  71.00  60.07   2659   73.488     4    0.056   73.544  0.00   4.65  74.18
 71.00  60.07   2659   73.488     4    0.056   73.544  0.00   4.65  74.29  71.00
 60.07   2659   73.488     4    0.056   73.544  0.00   4.65  74.29  71.00  60.07
  2659   73.488     4    0.056   73.544  0.00   4.65  74.29  71.00  60.07   2659
  73.488     4    0.056   73.544  0.00   4.65  74.29  71.00  60.07   2659   73.488
    4    0.056   73.544  0.00   4.65  74.29  71.00  60.07   2659   73.488    
4    0.056   73.544  0.00   4.65  74.29  71.00  60.07   2659   73.488     4  
 0.056   73.544  0.00   4.65  74.29  71.00  60.07   2659   73.488     4    0.056
  73.544  0.00   5.43  12.64  71.09  60.07   2661   73.534     4    0.056   73.590 
0.00   5.43  18.02  71.09  60.07   2661   73.534     4    0.056   73.590  0.00
  5.43  18.02  71.09  60.07   2661   73.534     4    0.056   73.590  0.00   5.43
 18.02  71.09  60.07   2661   73.534     4    0.056   73.590  0.00   5.43  18.02
 71.09  60.07   2661   73.534     4    0.056   73.590  0.00   5.43  18.02  71.09
 60.07   2661   73.534     4    0.056   73.590  0.00   5.43  18.02  71.09  60.07
  2661   73.534     4    0.056   73.590  0.00   5.43  18.02  71.09  60.07   2661
  73.534     4    0.056   73.590  0.00   5.43  18.02  71.09  60.07   2661   73.534
    4    0.056   73.590  0.00   5.43  18.02  71.09  60.07   2661   73.534    
4    0.056   73.590  0.00   5.43  18.02  71.09  60.07   2661   73.534     4  
 0.056   73.590  0.00   5.43  18.02  71.09  60.07   2661   73.534     4    0.056
  73.590  0.00   5.43  18.02  71.09  60.07   2661   73.534     4    0.056   73.590 
0.00   5.43  18.02  71.09  60.07   2661   73.534     4    0.056   73.590  0.00
  5.43  18.02  71.09  60.07   2661   73.534     4    0.056   73.590  0.00   5.43
 18.02  71.09  60.07   2661   73.534     4    0.056   73.590  0.00   5.43  18.02
 71.09  60.07   2661   73.534     4    0.056   73.590  0.00   5.43  18.02  71.09
 60.07   2661   73.534     4    0.056   73.590  0.00   5.43  69.24  71.09  60.07
  2661   73.534     4    0.056   73.590  0.00   5.43  78.05  71.09  60.07   2661
  73.534     4    0.056   73.590  0.00   5.43  78.97  71.09  60.07   2661   73.534
    4    0.056   73.590  0.00   5.43  79.07  71.09  60.07   2661   73.534    
4    0.056   73.590  0.00   5.43  79.18  71.09  60.07   2661   73.534     4  
 0.056   73.590  0.00   5.43  80.09  71.09  60.07   2661   73.534     4    0.056
  73.590  0.00   5.43  80.36  71.09  60.07   2661   73.534     4    0.056   73.590 
0.00   5.43  80.51  71.09  60.07   2661   73.534     4    0.056   73.590  0.00
  5.43  80.70  71.09  60.07   2661   73.534     4    0.056   73.590  0.00   5.43
 81.31  71.09  60.07   2661   73.534     4    0.056   73.590  0.00   5.43  81.63
 71.09  60.07   2661   73.534     4    0.056   73.590  0.00   5.43  82.94  71.09
 60.07   2661   73.534     4    0.056   73.590  0.00   5.43  82.94  71.09  60.07
  2661   73.534     4    0.056   73.590  0.00   5.43  82.94  71.09  60.07   2661
  73.534     4    0.056   73.590  0.00   5.43  82.94  71.09  60.07   2661   73.534
    4    0.056   73.590  0.00   5.43  82.94  71.09  60.07   2661   73.534    
4    0.056   73.590  0.00   5.43  82.94  71.09  60.07   2661   73.534     4  
 0.056   73.590  0.00   5.43  82.94  71.09  60.07   2661   73.534     4    0.056
  73.590  0.00   9.37  65.64  71.21  60.07   2663   73.583     4    0.056   73.639 
0.00   3.49  11.89  71.25  60.10   2665   73.623     4    0.056   73.679  0.00
  3.49  66.11  71.25  60.10   2665   73.623     4    0.056   73.679 14.26   0.00
  9.91  71.25  60.10   2666   73.649     4    0.056   73.705 14.26   0.00  73.25
 71.25  60.10   2666   73.649     4    0.056   73.705  0.00  10.49  14.43  71.29
 60.10   2667   73.675     4    0.056   73.731  0.00  10.49  66.07  71.29  60.10
  2667   73.675     4    0.056   73.731  0.00  10.49 100.00  71.29  60.10   2667
  73.675     4    0.056   73.731  0.00  10.49 100.00  71.29  60.10   2667   73.675
    4    0.056   73.731  0.00  10.49 100.00  71.29  60.10   2667   73.675    
4    0.056   73.731  0.00  10.49 100.00  71.29  60.10   2667   73.675     4  
 0.056   73.731  0.00  10.49 100.00  71.29  60.10   2667   73.675     4    0.056
  73.731  0.00  10.49 100.00  71.29  60.10   2667   73.675     4    0.056   73.731 
0.00  10.49 100.00  71.29  60.10   2667   73.675     4    0.056   73.731  0.00
 10.49 100.00  71.29  60.10   2667   73.675     4    0.056   73.731  0.00  10.49
100.00  71.29  60.10   2667   73.675     4    0.056   73.731  0.00  10.49 100.00
 71.29  60.10   2667   73.675     4    0.056   73.731  0.00  10.49 100.00  71.29
 60.10   2667   73.675     4    0.056   73.731  0.00  10.49 100.00  71.29  60.10
  2667   73.675     4    0.056   73.731  0.00  10.49 100.00  71.29  60.10   2667
  73.675     4    0.056   73.731  0.00  10.49 100.00  71.29  60.10   2667   73.675
    4    0.056   73.731  0.00  10.49 100.00  71.29  60.10   2667   73.675    
4    0.056   73.731

thanks.
jason


   
Mime
View raw message