Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Subject: Re: Cassandra 3.2.1: Memory leak?
To: user@cassandra.apache.org
References: 
 <CAEU_gmciZVa0Eb1ptvfo1PvBpztq0LpyycX=gYBNdNBXKqUivQ@mail.gmail.com>
From: "ssivikt@gmail.com" <ssivikt@gmail.com>
Message-ID: <56E49822.6010304@gmail.com>
Date: Sun, 13 Mar 2016 01:28:50 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: 
 <CAEU_gmciZVa0Eb1ptvfo1PvBpztq0LpyycX=gYBNdNBXKqUivQ@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------040709060008070809030005"

This is a multi-part message in MIME format.
--------------040709060008070809030005
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi, I'll duplicate here my email with the same issue

"/I have 7 nodes of C* v2.2.5 running on CentOS 7 and using jemalloc for 
dynamic storage allocation.
Use only one keyspace and one table with Leveled compaction strategy.
I've loaded ~500 GB of data into the cluster with replication factor 
equals to 3 and waiting until compaction is finished. But during 
compaction each of the C* nodes allocates all the available memory 
(~128GB) and just stops its process.
This is a known bug ? /"

On 03/13/2016 12:56 AM, Mohamed Lrhazi wrote:
> Hello,
>
> We installed Datastax community edition, on 8 nodes, RHEL7. We 
> inserted some 7 billion rows into a pretty simple table. the inserts 
> seem to have completed without issues. but ever since, we find that 
> the nodes reliably run out of RAM after few hours, without any user 
> activity at all. No reads nor write are sent at all.  What should we 
> look for to try and identify root cause?
>
>
> [root@avesterra-prod-1 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> [root@avesterra-prod-1 ~]# rpm -qa| grep datastax
> datastax-ddc-3.2.1-1.noarch
> datastax-ddc-tools-3.2.1-1.noarch
> [root@avesterra-prod-1 ~]#
>
> The nodes had 8 GB RAM, which we doubled twice and now are trying with 
> 40GB... they still manage to consume it all and cause oom_killer to 
> kick in.
>
> Pretty much all the settings are the default ones the installation 
> created.
>
> Thanks,
> Mohamed.

-- 
Thanks,
Serj


--------------040709060008070809030005
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Hi, I'll duplicate here my email with the same issue<br>
    <br>
    "<i><small>I have 7 nodes of C* v2.2.5 running on CentOS 7 and using
        jemalloc for dynamic storage allocation.
        <br>
        Use only one keyspace and one table with Leveled compaction
        strategy.
        <br>
        I've loaded ~500 GB of data into the cluster with replication
        factor equals to 3 and waiting until compaction is finished. But
        during compaction each of the C* nodes allocates all the
        available memory (~128GB) and just stops its process.
        <br>
        This is a known bug ?
      </small></i>"<br>
    <br>
    <div class="moz-cite-prefix">On 03/13/2016 12:56 AM, Mohamed Lrhazi
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAEU_gmciZVa0Eb1ptvfo1PvBpztq0LpyycX=gYBNdNBXKqUivQ@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>Hello,</div>
        <div><br>
        </div>
        <div>We installed Datastax community edition, on 8 nodes, RHEL7.
          We inserted some 7 billion rows into a pretty simple table.
          the inserts seem to have completed without issues. but ever
          since, we find that the nodes reliably run out of RAM after
          few hours, without any user activity at all. No reads nor
          write are sent at all.  What should we look for to try and
          identify root cause?</div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div>[root@avesterra-prod-1 ~]# cat /etc/redhat-release </div>
        <div>Red Hat Enterprise Linux Server release 7.2 (Maipo)</div>
        <div>[root@avesterra-prod-1 ~]# rpm -qa| grep datastax</div>
        <div>datastax-ddc-3.2.1-1.noarch</div>
        <div>datastax-ddc-tools-3.2.1-1.noarch</div>
        <div>[root@avesterra-prod-1 ~]# </div>
        <div><br>
        </div>
        <div>The nodes had 8 GB RAM, which we doubled twice and now are
          trying with 40GB... they still manage to consume it all and
          cause oom_killer to kick in.</div>
        <div><br>
        </div>
        <div>Pretty much all the settings are the default ones the
          installation created.</div>
        <div><br>
        </div>
        <div>Thanks,</div>
        <div>Mohamed.</div>
      </div>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="72">-- 
Thanks,
Serj</pre>
  </body>
</html>

--------------040709060008070809030005--