cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laszlo Szabo <laszlo.viktor.sz...@gmail.com>
Subject Bootstrap OOM issues with Cassandra 3.11.1
Date Mon, 06 Aug 2018 22:49:52 GMT
 Hello All,

I'm having JVM unstable / OOM errors when attempting to auto bootstrap a
9th node to an existing 8 node cluster (256 tokens).  Each machine has 24
cores 148GB RAM and 10TB (2TB used).  Under normal operation the 8 nodes
have JVM memory configured with Xms35G and Xmx35G, and handle 2-4 billion
inserts per day.  There are never updates, deletes, or sparsely populated
rows.

For the bootstrap node, I've tried memory values from 35GB to 135GB in 10GB
increments. I've tried using both memtable_allocation_types (heap_buffers
and offheap_buffers).  I've not tried modifying the
memtable_cleanup_threshold but instead have tried memtable_flush_writers
from 2 to 8.  I've tried memtable_(off)heap_space_in_mb from 20000 to
60000.  I've tried both CMS and G1 garbage collection with various
settings.

Typically, after streaming about ~2TB of data, CPU load will hit a maximum,
and the "nodetool info" heap memory will, over the course of an hour,
approach the maximum.  At that point, CPU load will drop to a single thread
with minimal activity until the system becomes unstable and eventually the
OOM error occurs.

Excerpt of the system log is below, and what I consistently see is the
MemtableFlushWriter and the MemtableReclaimMemory pending queues grow as
the memory becomes depleted, but the number of completed seems to stop
changing a few minutes after the CPU load spikes.

One other data point is there seems to be a huge number of mutations that
occur after most of the stream has occured.  Concurrent_writes is set at
256 with the queue getting as high as 200K before dropping down.

Any suggestions for yaml changes or jvm changes?  JVM.options is currently
the default with the memory set to the max, the current YAML file is below.

Thanks!


INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,329 StatusLogger.java:51 -
> MutationStage                     1         2      191498052         0
>            0
> INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,331 StatusLogger.java:51 -
> ViewMutationStage                 0         0              0         0
>            0
> INFO  [Service Thread] 2018-08-06 17:49:26,338 StatusLogger.java:51 -
> PerDiskMemtableFlushWriter_0         0         0           5865         0
>                0
> INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,343 StatusLogger.java:51 -
> ReadStage                         0         0              0         0
>            0
> INFO  [Service Thread] 2018-08-06 17:49:26,347 StatusLogger.java:51 -
> ValidationExecutor                0         0              0         0
>            0
> INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,360 StatusLogger.java:51 -
> RequestResponseStage              0         0              8         0
>            0
> INFO  [Service Thread] 2018-08-06 17:49:26,380 StatusLogger.java:51 -
> Sampler                           0         0              0         0
>            0
> INFO  [Service Thread] 2018-08-06 17:49:26,382 StatusLogger.java:51 - *MemtableFlushWriter
>              8     74293           4716         0  *               0
> INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,388 StatusLogger.java:51 -
> ReadRepairStage                   0         0              0         0
>            0
> INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,389 StatusLogger.java:51 -
> CounterMutationStage              0         0              0         0
>            0
> INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,404 StatusLogger.java:51 -
> MiscStage                         0         0              0         0
>            0
> INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,407 StatusLogger.java:51 -
> CompactionExecutor                8        13            493         0
>            0
> INFO  [Service Thread] 2018-08-06 17:49:26,410 StatusLogger.java:51 -
> InternalResponseStage             0         0             16         0
>            0
> INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,413 StatusLogger.java:51 - *MemtableReclaimMemory
>            1      6066            356         0     *            0
> INFO  [Service Thread] 2018-08-06 17:49:26,421 StatusLogger.java:51 -
> AntiEntropyStage                  0         0              0         0
>            0
> INFO  [Service Thread] 2018-08-06 17:49:26,430 StatusLogger.java:51 -
> CacheCleanupExecutor              0         0              0         0
>            0
> INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,431 StatusLogger.java:51 -
> PendingRangeCalculator            0         0              9         0
>            0
> INFO  [Service Thread] 2018-08-06 17:49:26,436 StatusLogger.java:61 -
> CompactionManager                 8        19




 Current Yaml

num_tokens: 256

hinted_handoff_enabled: true

hinted_handoff_throttle_in_kb: 10240

max_hints_delivery_threads: 8

hints_flush_period_in_ms: 10000

max_hints_file_size_in_mb: 128

batchlog_replay_throttle_in_kb: 10240

authenticator: AllowAllAuthenticator

authorizer: AllowAllAuthorizer

role_manager: CassandraRoleManager

roles_validity_in_ms: 2000

permissions_validity_in_ms: 2000

credentials_validity_in_ms: 2000

partitioner: org.apache.cassandra.dht.Murmur3Partitioner

data_file_directories:

    - /data/cassandra/data

commitlog_directory: /data/cassandra/commitlog

cdc_enabled: false

disk_failure_policy: stop

commit_failure_policy: stop

prepared_statements_cache_size_mb:

thrift_prepared_statements_cache_size_mb:

key_cache_size_in_mb:

key_cache_save_period: 14400

row_cache_size_in_mb: 0

row_cache_save_period: 0

counter_cache_size_in_mb:

counter_cache_save_period: 7200

saved_caches_directory: /data/cassandra/saved_caches

commitlog_sync: periodic

commitlog_sync_period_in_ms: 10000

commitlog_segment_size_in_mb: 32

seed_provider:

    - class_name: org.apache.cassandra.locator.SimpleSeedProvider

      parameters:

          - seeds: "10.1.1.11,10.1.1.12,10.1.1.13"

concurrent_reads: 128

concurrent_writes: 256

concurrent_counter_writes: 96

concurrent_materialized_view_writes: 32

disk_optimization_strategy: spinning

memtable_heap_space_in_mb: 61440

memtable_offheap_space_in_mb: 61440

memtable_allocation_type: heap_buffers

commitlog_total_space_in_mb: 81920

memtable_flush_writers: 8

Mime
View raw message