cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Feng Qu <mail...@gmail.com>
Subject Re: nodetool ring runs very slow
Date Tue, 27 Mar 2012 17:03:46 GMT
Hi Jonathan, similar problem happens again and there is only one GC running at that time per
system.log. 

This is one node of a 6-node 0.8.6 ring. Heap size on this host is 16GB.

[fengqu@slcdbx1035 cassandra]$ date; time cnt ring
Tue Mar 27 09:24:20 GMT+7 2012
Address        
DC         
Rack        Status State  
Load           
Owns   
Token                                       
                                                                               141784319550391026443072753096570088106    

10.89.74.60    
slc        
rack0       Up    
Normal  343.97 GB       16.67% 
0                                           
10.2.128.55    
phx        
rack0       Up    
Normal  485.48 GB       16.67% 
28356863910078205288614550619314017621      
10.89.74.67    
slc        
rack0       Up    
Normal  252.82 GB       16.67% 
56713727820156410577229101238628035242      
10.2.128.56    
phx        
rack0       Up    
Normal  258.49 GB       16.67% 
85070591730234615865843651857942052864      
10.89.74.62     slc    
    rack0      
Up     Normal  251.02
GB       16.67%  113427455640312821154458202477256070485     
10.2.128.57     phx        
rack0       Up    
Normal  451.97 GB       16.67% 
141784319550391026443072753096570088106     
 
real   
0m25.999s
user    0m0.309s
sys     0m0.048s

  WARN [ScheduledTasks:1] 2012-03-27 09:22:39,499 GCInspector.java (line 143) Heap is 0.8716646568744459
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up
to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 INFO [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
1383 ms for 2 collections, 14756794512 used; max is 16928210944
 WARN [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 143) Heap is 0.8717279434204102
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up
to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 INFO [ScheduledTasks:1] 2012-03-27 09:24:04,844 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
3090 ms for 2 collections, 14782314472 used; max is 16928210944
 WARN [ScheduledTasks:1] 2012-03-27 09:24:04,851 GCInspector.java (line 143) Heap is 0.8732354837083013
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up
to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 INFO [ScheduledTasks:1] 2012-03-27 09:24:46,610 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
3057 ms for 2 collections, 14757982328 used; max is 16928210944
 WARN [ScheduledTasks:1] 2012-03-27 09:24:46,616 GCInspector.java (line 143) Heap is 0.871798111260587
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up
to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 INFO [ScheduledTasks:1] 2012-03-27 09:25:36,371 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
7430 ms for 2 collections, 14719911032 used; max is 16928210944
 WARN [ScheduledTasks:1] 2012-03-27 09:25:36,377 GCInspector.java (line 143) Heap is 0.8695491260532345
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up
to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 
Feng Qu


>________________________________
> From: Jonathan Ellis <jbellis@gmail.com>
>To: user <user@cassandra.apache.org> 
>Cc: Feng Qu <mail4qf@gmail.com> 
>Sent: Friday, February 24, 2012 2:29 PM
>Subject: Re: nodetool ring runs very slow
> 
>Read the server log and look for GCInspector output.
>
>On Fri, Feb 24, 2012 at 11:02 AM, Feng Qu <mail4qf@gmail.com> wrote:
>> Hi Jonathan, how to check out whether it's in GC storming? This server
>> crashed few time due to Java heap out of memory. We use 8GB heap on a server
>> with 96GB ram. This is first node in a 6-node ring and it has opscenter
>> community running on it.
>>
>> Feng Qu
>>
>> ________________________________
>> From: Jonathan Ellis <jbellis@gmail.com>
>> To: user@cassandra.apache.org; Feng Qu <mail4qf@gmail.com>
>> Sent: Thursday, February 23, 2012 1:19 PM
>> Subject: Re: nodetool ring runs very slow
>>
>> The only time I've seen nodetool be that slow is when it was talking
>> to a machine that was either swapping or deep into (JVM) GC storming.
>>
>> On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu <mail4qf@gmail.com> wrote:
>>> We noticed that nodetool ring sometimes returns in 17-20 sec while it
>>> normally runs in less than a sec. There were some compaction running when
>>> it
>>> happened. Did compaction cause nodetool slowness? Anything else I should
>>> check?
>>>>>> time nodetool -h hostname ring
>>> ....
>>> real 0m17.595s
>>> user 0m0.339s
>>> sys 0m0.054s
>>>
>>> Feng Qu
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>>
>
>
>
>-- 
>Jonathan Ellis
>Project Chair, Apache Cassandra
>co-founder of DataStax, the source for professional Cassandra support
>http://www.datastax.com
>
>
>
Mime
View raw message