incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haithem Jarraya <haithem.jarr...@struq.com>
Subject Re: Advice on memory warning
Date Tue, 23 Apr 2013 16:47:39 GMT
We are facing similar issue, and we are not able to have the ring stable.
 We are using C*1.2.3 on Centos6, 32GB - RAM, 8GB-heap, 6 Nodes.
The total data ~ 84gb (which is relatively small for C* to handle, with a
RF of 3).  Our application is heavy read, we see the GC complaints in all
nodes, I copied and past the output below.
Also we usually see much larger values for the Pending - ReadStage, not
sure what is the best advice for this.

Thanks,

Haithem

INFO [ScheduledTasks:1] 2013-04-23 16:40:02,118 GCInspector.java (line 119)
GC for ConcurrentMarkSweep: 911 ms for 1 collections, 5945542968 used; max
is 8199471104
 INFO [ScheduledTasks:1] 2013-04-23 16:40:16,051 GCInspector.java (line
119) GC for ConcurrentMarkSweep: 322 ms for 1 collections, 5639896576 used;
max is 8199471104
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,829 GCInspector.java (line
119) GC for ConcurrentMarkSweep: 2273 ms for 1 collections, 6762618136
used; max is 8199471104
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,830 StatusLogger.java (line
53) Pool Name                    Active   Pending   Blocked
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,830 StatusLogger.java (line
68) ReadStage                         4         4         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,831 StatusLogger.java (line
68) RequestResponseStage              1         6         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,831 StatusLogger.java (line
68) ReadRepairStage                   0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,831 StatusLogger.java (line
68) MutationStage                     0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,831 StatusLogger.java (line
68) ReplicateOnWriteStage             0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,832 StatusLogger.java (line
68) GossipStage                       0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,832 StatusLogger.java (line
68) AntiEntropyStage                  0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,832 StatusLogger.java (line
68) MigrationStage                    0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,832 StatusLogger.java (line
68) MemtablePostFlusher               0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,833 StatusLogger.java (line
68) FlushWriter                       0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,833 StatusLogger.java (line
68) MiscStage                         0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,833 StatusLogger.java (line
68) commitlog_archiver                0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,834 StatusLogger.java (line
68) InternalResponseStage             0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,834 StatusLogger.java (line
68) AntiEntropySessions               0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,834 StatusLogger.java (line
68) HintedHandoff                     0         0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,843 StatusLogger.java (line
73) CompactionManager                 0         0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,844 StatusLogger.java (line
85) MessagingService                n/a      15,1
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,844 StatusLogger.java (line
95) Cache Type                     Size                 Capacity
    KeysToSave
Provider
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,844 StatusLogger.java (line
96) KeyCache                  251658064                251658081
           all
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,844 StatusLogger.java (line
102) RowCache                          0                        0
           all
 org.apache.cassandra.cache.SerializingCacheProvider
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,844 StatusLogger.java (line
109) ColumnFamily                Memtable ops,data
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,845 StatusLogger.java (line
112) system.local                              0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,845 StatusLogger.java (line
112) system.peers                              0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,845 StatusLogger.java (line
112) system.batchlog                           0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,845 StatusLogger.java (line
112) system.NodeIdInfo                         0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,846 StatusLogger.java (line
112) system.LocationInfo                       0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,846 StatusLogger.java (line
112) system.Schema                             0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,846 StatusLogger.java (line
112) system.Migrations                         0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,846 StatusLogger.java (line
112) system.schema_keyspaces                   0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,846 StatusLogger.java (line
112) system.schema_columns                     0,0
INFO [ScheduledTasks:1] 2013-04-23 16:40:30,846 StatusLogger.java (line
112) system.schema_columnfamilies                 0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,847 StatusLogger.java (line
112) system.IndexInfo                          0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,847 StatusLogger.java (line
112) system.range_xfers                        0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,847 StatusLogger.java (line
112) system.peer_events                        0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,847 StatusLogger.java (line
112) system.hints                              0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,847 StatusLogger.java (line
112) system.HintsColumnFamily                  0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,848 StatusLogger.java (line
112) x.foo                 0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,848 StatusLogger.java (line
112) x.foo2                 0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,848 StatusLogger.java (line
112) x.foo3                 0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,848 StatusLogger.java (line
112) x.foo4                 0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,848 StatusLogger.java (line
112) x.foo5                      0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,849 StatusLogger.java (line
112) x.foo6                 0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,849 StatusLogger.java (line
112) x.foo7                     0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,849 StatusLogger.java (line
112) system_auth.users                         0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,849 StatusLogger.java (line
112) system_traces.sessions                    0,0
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,849 StatusLogger.java (line
112) system_traces.events                      0,0
 WARN [ScheduledTasks:1] 2013-04-23 16:40:30,850 GCInspector.java (line
142) Heap is 0.824762725573964 full.  You may need to reduce memtable
and/or cache sizes.  Cassandra will now flush up to the two largest
memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
 INFO [ScheduledTasks:1] 2013-04-23 16:40:30,850 StorageService.java (line
3537) Unable to reduce heap usage since there are no dirty column families




On 23 April 2013 16:52, Ralph Goers <ralph.goers@dslextreme.com> wrote:

> We are using DSE, which I believe is also 1.1.9.  We have basically had a
> non-usable cluster for months due to this error.  In our case, once it
> starts doing this it starts flushing sstables to disk and eventually fills
> up the disk to the point where it can't compact.  If we catch it soon
> enough and restart the node it usually can recover.
>
> In our case, the heap size is 12 GB. As I understand it Cassandra will
> give 1/3 of that for sstables. I then noticed that we have one column
> family that is using nearly 4GB in bloom filters on each node.  Since the
> nodes will start doing this when the heap reaches 9GB we essentially only
> have 1GB of free memory so when compactions, cleanups, etc take place this
> situation starts happening.  We are working to change our data model to try
> to resolve this.
>
> Ralph
>
> On Apr 19, 2013, at 8:00 AM, Michael Theroux wrote:
>
> > Hello,
> >
> > We've recently upgraded from m1.large to m1.xlarge instances on AWS to
> handle additional load, but to also relieve memory pressure.  It appears to
> have accomplished both, however, we are still getting a warning, 0-3 times
> a day, on our database nodes:
> >
> > WARN [ScheduledTasks:1] 2013-04-19 14:17:46,532 GCInspector.java (line
> 145) Heap is 0.7529240824406468 full.  You may need to reduce memtable
> and/or cache sizes.  Cassandra will now flush up to the two largest
> memtables to free up memory.  Adjust flush_largest_memtables_at threshold
> in cassandra.yaml if you don't want Cassandra to do this automatically
> >
> > This is happening much less frequently than before the upgrade, but
> after essentially doubling the amount of available memory, I'm curious on
> what I can do to determine what is happening during this time.
> >
> > I am collecting all the JMX statistics.  Memtable space is elevated but
> not extraordinarily high.  No GC messages are being output to the log.
> >
> > These warnings do seem to be occurring doing compactions of column
> families using LCS with wide rows, but I'm not sure there is a direct
> correlation.
> >
> > We are running Cassandra 1.1.9, with a maximum heap of 8G.
> >
> > Any advice?
> > Thanks,
> > -Mike
>
>

Mime
View raw message