cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: nodetool ring runs very slow
Date Fri, 30 Mar 2012 23:48:01 GMT
You node is under significant memory pressure. 

Look into:
* reducing caches
* reducing the JVM heap to 8GB

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 28/03/2012, at 6:03 AM, Feng Qu wrote:

> Hi Jonathan, similar problem happens again and there is only one GC running at that time
per system.log. 
> 
> This is one node of a 6-node 0.8.6 ring. Heap size on this host is 16GB.
> 
> [fengqu@slcdbx1035 cassandra]$ date; time cnt ring
> Tue Mar 27 09:24:20 GMT+7 2012
> Address         DC          Rack        Status State   Load            Owns    Token
                                     
>                                                                                141784319550391026443072753096570088106
   
> 10.89.74.60     slc         rack0       Up     Normal  343.97 GB       16.67%  0    
                                     
> 10.2.128.55     phx         rack0       Up     Normal  485.48 GB       16.67%  28356863910078205288614550619314017621
    
> 10.89.74.67     slc         rack0       Up     Normal  252.82 GB       16.67%  56713727820156410577229101238628035242
    
> 10.2.128.56     phx         rack0       Up     Normal  258.49 GB       16.67%  85070591730234615865843651857942052864
    
> 10.89.74.62     slc         rack0       Up     Normal  251.02 GB       16.67%  113427455640312821154458202477256070485
   
> 10.2.128.57     phx         rack0       Up     Normal  451.97 GB       16.67%  141784319550391026443072753096570088106
   
>  
> real    0m25.999s
> user    0m0.309s
> sys     0m0.048s
> 
>   WARN [ScheduledTasks:1] 2012-03-27 09:22:39,499 GCInspector.java (line 143) Heap is
0.8716646568744459 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will
now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>  INFO [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
1383 ms for 2 collections, 14756794512 used; max is 16928210944
>  WARN [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 143) Heap is
0.8717279434204102 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will
now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>  INFO [ScheduledTasks:1] 2012-03-27 09:24:04,844 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
3090 ms for 2 collections, 14782314472 used; max is 16928210944
>  WARN [ScheduledTasks:1] 2012-03-27 09:24:04,851 GCInspector.java (line 143) Heap is
0.8732354837083013 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will
now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>  INFO [ScheduledTasks:1] 2012-03-27 09:24:46,610 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
3057 ms for 2 collections, 14757982328 used; max is 16928210944
>  WARN [ScheduledTasks:1] 2012-03-27 09:24:46,616 GCInspector.java (line 143) Heap is
0.871798111260587 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will
now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>  INFO [ScheduledTasks:1] 2012-03-27 09:25:36,371 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
7430 ms for 2 collections, 14719911032 used; max is 16928210944
>  WARN [ScheduledTasks:1] 2012-03-27 09:25:36,377 GCInspector.java (line 143) Heap is
0.8695491260532345 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will
now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>  
> Feng Qu
> From: Jonathan Ellis <jbellis@gmail.com>
> To: user <user@cassandra.apache.org> 
> Cc: Feng Qu <mail4qf@gmail.com> 
> Sent: Friday, February 24, 2012 2:29 PM
> Subject: Re: nodetool ring runs very slow
> 
> Read the server log and look for GCInspector output.
> 
> On Fri, Feb 24, 2012 at 11:02 AM, Feng Qu <mail4qf@gmail.com> wrote:
> > Hi Jonathan, how to check out whether it's in GC storming? This server
> > crashed few time due to Java heap out of memory. We use 8GB heap on a server
> > with 96GB ram. This is first node in a 6-node ring and it has opscenter
> > community running on it.
> >
> > Feng Qu
> >
> > ________________________________
> > From: Jonathan Ellis <jbellis@gmail.com>
> > To: user@cassandra.apache.org; Feng Qu <mail4qf@gmail.com>
> > Sent: Thursday, February 23, 2012 1:19 PM
> > Subject: Re: nodetool ring runs very slow
> >
> > The only time I've seen nodetool be that slow is when it was talking
> > to a machine that was either swapping or deep into (JVM) GC storming.
> >
> > On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu <mail4qf@gmail.com> wrote:
> >> We noticed that nodetool ring sometimes returns in 17-20 sec while it
> >> normally runs in less than a sec. There were some compaction running when
> >> it
> >> happened. Did compaction cause nodetool slowness? Anything else I should
> >> check?
> >>>>> time nodetool -h hostname ring
> >> ....
> >> real 0m17.595s
> >> user 0m0.339s
> >> sys 0m0.054s
> >>
> >> Feng Qu
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of DataStax, the source for professional Cassandra support
> > http://www.datastax.com
> >
> >
> 
> 
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
> 
> 


Mime
View raw message