Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Mon, 20 Jun 2016 19:06:57 +0000 (UTC)
From: "Thom Bartold (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12919449.1449326655000.1237.1466449617872@Atlassian.JIRA>
In-Reply-To: <JIRA.12919449.1449326655000@Atlassian.JIRA>
References: <JIRA.12919449.1449326655000@Atlassian.JIRA> <JIRA.12919449.1449326655590@arcas>
Subject: [jira] [Commented] (CASSANDRA-10821) OOM Killer terminates
 Cassandra when Compactions use too much memory then won't restart
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 20 Jun 2016 19:06:59 -0000


    [ https://issues.apache.org/jira/browse/CASSANDRA-10821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15340195#comment-15340195 ] 

Thom Bartold commented on CASSANDRA-10821:
------------------------------------------

Months later, in a new environment we are running into the OOM problem again.

We have upgraded to version 2.2.5 so the 'saved_chache_directory' problem is gone.

Our new configuration is 24 i2-xlarge AWS instances in 3 Availability zones. Our main data set is about 1 billion records stored 3 times across the cluster, using EC2 Snitch and Network topology 3 x us-east-1.

After an initial 'repair -pr' that ran from June 14 to June 17:

INFO  [Thread-5215] 2016-06-14 19:59:02,938 RepairRunnable.java:124 - Starting repair command #1, repairing keyspace overlordprod with repair options (parallelism: parallel, primary range: true, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 256)
INFO  [CompactionExecutor:663] 2016-06-17 01:59:25,742 RepairRunnable.java:309 - Repair command #1 finished in 2 days 6 hours 0 minutes 22 seconds
INFO  [Thread-10147] 2016-06-17 01:59:25,853 RepairRunnable.java:124 - Starting repair command #2, repairing keyspace system_traces with repair options (parallelism: parallel, primary range: true, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 256)
INFO  [InternalResponseStage:70] 2016-06-17 02:25:40,721 RepairRunnable.java:309 - Repair command #2 finished in 26 minutes 14 seconds

The number of compactions has gone up and down in the range 800-1500, but is now staying at around 800.

In the last two days, Cassandra has been OOM killed twice:

Jun 19 15:22:27 ip-10-242-145-240 kernel: [5329487.278454] Killed process 16391 (java) total-vm:749879160kB, anon-rss:28776424kB, file-rss:85040kB
Jun 20 07:36:18 ip-10-242-145-240 kernel: [5387918.072086] Killed process 15716 (java) total-vm:766699696kB, anon-rss:28955732kB, file-rss:118336kB

We believe it is because it is working on one really big compaction:

pending tasks: 811
                                     id   compaction type       keyspace      table      completed          total    unit   progress
   c8ff1fb0-3716-11e6-917f-93c4050661a3        Compaction   overlordprod   document     4875839081     6753090774   bytes     72.20%
   f3344ec0-36b9-11e6-917f-93c4050661a3        Compaction   overlordprod   document   225617209930   436848813618   bytes     51.65%
   fcec5120-3717-11e6-917f-93c4050661a3        Compaction   overlordprod   document     2025919777    16765177436   bytes     12.08%
   d2a960b0-3717-11e6-917f-93c4050661a3        Compaction   overlordprod   document     2459846227     6033377128   bytes     40.77%
Active compaction remaining time :   3h49m53s

We should be able to verify that the same compaction does not complete after the next OOM kill.

Note that the i2-xlarge instances have only 800GB of disk, and this one is currently 85% full which may be part o the problem. Other nodes in the cluster are not using more than 70%.

Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/xvdb      781029612 660898356 120131256  85% /data


> OOM Killer terminates Cassandra when Compactions use too much memory then won't restart
> ---------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-10821
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10821
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: EC2 32 x i2.xlarge split between us-east-1a,c and us-west 2a,b
> Linux  4.1.10-17.31.amzn1.x86_64 #1 SMP Sat Oct 24 01:31:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
> Cassandra version: 2.2.3
>            Reporter: Thom Bartold
>
> We were writing to the DB from EC2 instances in us-east-1 at a rate of about 3000 per second, replication us-east:2 us-west:2, LeveledCompaction and DeflateCompressor.
> After about 48 hours some nodes had over 800 pending compactions and a few of them started getting killed for Linux OOM. Priam attempts to restart the nodes, but they fail because of corrupted saved_cahce files.
> Loading has finished, and the cluster is mostly idle, but 6 of the nodes were killed again last night by OOM.
> This is the log message where the node won't restart:
> ERROR [main] 2015-12-05 13:59:13,754 CassandraDaemon.java:635 - Detected unreadable sstables /media/ephemeral0/cassandra/saved_caches/KeyCache-ca.db, please check NEWS.txt and ensure that you have upgraded through all required intermediate versions, running upgradesstables
> This is the dmesg where the node is terminated:
> [360803.234422] Out of memory: Kill process 10809 (java) score 949 or sacrifice child
> [360803.237544] Killed process 10809 (java) total-vm:438484092kB, anon-rss:29228012kB, file-rss:107576kB
> This is what Compaction Stats look like currently:
> pending tasks: 1096
>                                      id   compaction type          keyspace      table    completed          total    unit   progress
>    93eb3200-9b58-11e5-b9f1-ffef1041ec45        Compaction   overlordpreprod   document   8670748796   839129219651   bytes      1.03%
>                                                Compaction            system      hints           30     1921326518   bytes      0.00%
> Active compaction remaining time :  27h33m47s
> Only 6 of the 32 nodes have compactions pending, and all on the order of 1000.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)