cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyrylo Lebediev <klebed...@conductor.com>
Subject Re: Maximum memory usage reached
Date Thu, 07 Mar 2019 11:55:59 GMT
Got it.
Thank you for helping me, Jon, Jeff!

> Is there a reason why you’re picking Cassandra for this dataset?
Decision wasn’t made by myself, I guess C* was chosen because some huge growth was planned.

Regards,
Kyrill

From: Jeff Jirsa <jjirsa@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wednesday, March 6, 2019 at 22:19
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Maximum memory usage reached

Also, that particular logger is for the internal chunk / page cache. If it can’t allocate
from within that pool, it’ll just use a normal bytebuffer.

It’s not really a problem, but if you see performance suffer, upgrade to latest 3.11.4,
there was a bit of a perf improvement in the case where that cache fills up.

--
Jeff Jirsa


On Mar 6, 2019, at 11:40 AM, Jonathan Haddad <jon@jonhaddad.com<mailto:jon@jonhaddad.com>>
wrote:
That’s not an error. To the left of the log message is the severity, level INFO.

Generally, I don’t recommend running Cassandra on only 2GB ram or for small datasets that
can easily fit in memory. Is there a reason why you’re picking Cassandra for this dataset?

On Thu, Mar 7, 2019 at 8:04 AM Kyrylo Lebediev <klebediev@conductor.com<mailto:klebediev@conductor.com>>
wrote:
Hi All,

We have a tiny 3-node cluster
C* version 3.9 (I know 3.11 is better/stable, but can’t upgrade immediately)
HEAP_SIZE is 2G
JVM options are default
All setting in cassandra.yaml are default (file_cache_size_in_mb not set)

Data per node – just ~ 1Gbyte

We’re getting following errors messages:

DEBUG [CompactionExecutor:87412] 2019-03-06 11:00:13,545 CompactionTask.java:150 - Compacting
(ed4a4d90-4028-11e9-adc0-230e0d6622df) [/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-23248-big-Data.db:level=0,
/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-23247-big-Data.db:level=0,
/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-23246-big-Data.db:level=0,
/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-23245-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:87412] 2019-03-06 11:00:13,582 CompactionTask.java:230 - Compacted
(ed4a4d90-4028-11e9-adc0-230e0d6622df) 4 sstables to [/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-23249-big,]
to level=0.  6.264KiB to 1.485KiB (~23% of original) in 36ms.  Read Throughput = 170.754KiB/s,
Write Throughput = 40.492KiB/s, Row Throughput = ~106/s.  194 total partitions merged to 44.
 Partition merge counts were {1:18, 4:44, }
INFO  [IndexSummaryManager:1] 2019-03-06 11:00:22,007 IndexSummaryRedistribution.java:75 -
Redistributing index summaries
INFO  [pool-1-thread-1] 2019-03-06 11:11:24,903 NoSpamLogger.java:91 - Maximum memory usage
reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO  [pool-1-thread-1] 2019-03-06 11:26:24,926 NoSpamLogger.java:91 - Maximum memory usage
reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO  [pool-1-thread-1] 2019-03-06 11:41:25,010 NoSpamLogger.java:91 - Maximum memory usage
reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO  [pool-1-thread-1] 2019-03-06 11:56:25,018 NoSpamLogger.java:91 - Maximum memory usage
reached (512.000MiB), cannot allocate chunk of 1.000MiB

What’s interesting that “Maximum memory usage reached” messages appears each 15 minutes.
Reboot temporary solve the issue, but it then appears again after some time

Checked, there are no huge partitions (max partition size is ~2Mbytes )

How such small amount of data may cause this issue?
How to debug this issue further?


Regards,
Kyrill


--
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade
Mime
View raw message