Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Thu, 5 Oct 2017 08:28:03 +0000 (UTC)
From: "Thomas Steinmaurer (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.13106635.1507017484000.277456.1507192083214@Atlassian.JIRA>
In-Reply-To: <JIRA.13106635.1507017484000@Atlassian.JIRA>
References: <JIRA.13106635.1507017484000@Atlassian.JIRA> <JIRA.13106635.1507017484033@jira-lw-us.apache.org>
Subject: [jira] [Updated] (CASSANDRA-13929) BTree$Builder /
 io.netty.util.Recycler$Stack leaking memory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 05 Oct 2017 08:28:10 -0000


     [ https://issues.apache.org/jira/browse/CASSANDRA-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Steinmaurer updated CASSANDRA-13929:
-------------------------------------------
    Attachment: cassandra_3.11.1_NORECYCLE_memory_utilization.jpg

Patched our 9 node loadtest cluster with disabled recycling. Flat AVG heap utilization in a time-frame of 24hrs. => cassandra_3.11.1_NORECYCLE_memory_utilization.jpg

I don't see any negative impact for normal operations (not streaming) running without this cache, so this is probably a cost vs. benefit discussion, but e.g. in relation to other in-memory data structures for speeding up things (key cache, bloom filter), this kind of cache is questionable, IMHO.


> BTree$Builder / io.netty.util.Recycler$Stack leaking memory
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-13929
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13929
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Thomas Steinmaurer
>         Attachments: cassandra_3.11.0_min_memory_utilization.jpg, cassandra_3.11.1_mat_dominator_classes_FIXED.png, cassandra_3.11.1_mat_dominator_classes.png, cassandra_3.11.1_NORECYCLE_memory_utilization.jpg, cassandra_3.11.1_snapshot_heaputilization.png
>
>
> Different to CASSANDRA-13754, there seems to be another memory leak in 3.11.0+ in BTree$Builder / io.netty.util.Recycler$Stack.
> * heap utilization increase after upgrading to 3.11.0 => cassandra_3.11.0_min_memory_utilization.jpg
> * No difference after upgrading to 3.11.1 (snapshot build) => cassandra_3.11.1_snapshot_heaputilization.png; thus most likely after fixing CASSANDRA-13754, more visible now
> * MAT shows io.netty.util.Recycler$Stack as top contributing class => cassandra_3.11.1_mat_dominator_classes.png
> * With -Xmx8G (CMS) and our load pattern, we have to do a rolling restart after ~ 72 hours
> Verified the following fix, namely explicitly unreferencing the _recycleHandle_ member (making it non-final). In _org.apache.cassandra.utils.btree.BTree.Builder.recycle()_
> {code}
>         public void recycle()
>         {
>             if (recycleHandle != null)
>             {
>                 this.cleanup();
>                 builderRecycler.recycle(this, recycleHandle);
>                 recycleHandle = null; // ADDED
>             }
>         }
> {code}
> Patched a single node in our loadtest cluster with this change and after ~ 10 hours uptime, no sign of the previously offending class in MAT anymore => cassandra_3.11.1_mat_dominator_classes_FIXED.png
> Can' say if this has any other side effects etc., but I doubt.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org