Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0CEFD200B27 for ; Wed, 22 Jun 2016 15:03:42 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 083ED160A35; Wed, 22 Jun 2016 13:03:42 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 028E3160A2E for ; Wed, 22 Jun 2016 15:03:40 +0200 (CEST) Received: (qmail 53257 invoked by uid 500); 22 Jun 2016 13:03:39 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 53247 invoked by uid 99); 22 Jun 2016 13:03:39 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2016 13:03:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 15DF5C1308 for ; Wed, 22 Jun 2016 13:03:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id h70aob_xCBPp for ; Wed, 22 Jun 2016 13:03:37 +0000 (UTC) Received: from mail-wm0-f49.google.com (mail-wm0-f49.google.com [74.125.82.49]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 767C85F4E3 for ; Wed, 22 Jun 2016 13:03:36 +0000 (UTC) Received: by mail-wm0-f49.google.com with SMTP id r201so5145531wme.1 for ; Wed, 22 Jun 2016 06:03:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=VGVGtF0u3fofsm4ImldUGg4ELCytcwqffBP17yBbouI=; b=f65+F2oiOxam5TdXUTRbcD7EFKp3E4YfO1vtOiyTQ/9+0aIzYSjr0WFM5Javjage4c nAhXS2Nb84iRXM0x4V4MlC48zsuJs0FxvmCyCI11SzZogGwc4UAwtyV2cVHote9umq5+ UuOcpHW1Ig4xAI3IT+vAdvwBr5ji5FN1B8Zh5lqGXk8EyLtr7aB768QD6DEJ16wsjZza qI1Zvb32eK3KBB8s01OAtEPb0fVLv+B86gghVKQZG0H9rnS52zYjRq6b6qeNjoAQ9Lqx E2lRIA3wmLqQTqwxtt7J+zdTvTmn6m19PB3+6Olx0AwSAhd2j54YmruGAkORMAvE2G3+ sZ2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=VGVGtF0u3fofsm4ImldUGg4ELCytcwqffBP17yBbouI=; b=CtVb5npCgGNLePKlq8LmP03ppY7WqPaZ8gAeFLQsvPXFIm4hJ+hqmRpFclXp8UL+Zt w8RwZ6XlCyAps4vyU+fYd7oayZEx7OcRSMADRdMw7PzauwFE9bjBwRFzlDNBExtiHSJZ sK6vcgEkoBoPBkiucYqVVKBKNL5hEFyNAM9FX/j+h8iZQqKrY+GRbqnAfuUOT5iJ+Loy tFSfqOf5p53/1ouZi4T3ymUpB5Z1zZeapeAoON5Xy4Pt2vV9Ps9CveoGLr2Zfapo37rg DR76SyaHZF8oVoutNX8qzIWLT3rWRUbae8KMCZ9XJP1FAL3PJZ5TofH82VUpe4RhGgsz WI9Q== X-Gm-Message-State: ALyK8tKpUAnLotmN8Dph1wpNm9HW0wbqW4GrMafmdbeJiERJzn7l+EetUUaS0IQu8FSnmi+5o4DkUInOjBCWgQ== X-Received: by 10.194.52.74 with SMTP id r10mr24471019wjo.50.1466600614785; Wed, 22 Jun 2016 06:03:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.53.131 with HTTP; Wed, 22 Jun 2016 06:03:34 -0700 (PDT) In-Reply-To: References: From: Marcus Eriksson Date: Wed, 22 Jun 2016 15:03:34 +0200 Message-ID: Subject: Re: High Heap Memory usage during nodetool repair in Cassandra 3.0.3 To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b8744f4ac03c60535dd9061 archived-at: Wed, 22 Jun 2016 13:03:42 -0000 --047d7b8744f4ac03c60535dd9061 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable it could also be CASSANDRA-11412 if you have many sstables and vnodes On Wed, Jun 22, 2016 at 2:50 PM, Bhuvan Rawal wrote: > Thanks for the info Paulo, Robert. I tried further testing with other > parameters and it was prevalent. We could be either 11739, 11206. But im > spektical about 11739 because repair works well in 3.5 and 11739 seems to > be fixed for 3.7/3.0.7. > > We may possibly resolve this by increasing heap size thereby reducing som= e > page cache bandwidth before upgrading to higher versions. > > On Mon, Jun 20, 2016 at 10:00 PM, Paulo Motta > wrote: > >> You could also be hitting CASSANDRA-11739, which was fixed on 3.0.7 and >> could potentially cause OOMs for long-running repairs. >> >> >> 2016-06-20 13:26 GMT-03:00 Robert Stupp : >> >>> One possibility might be CASSANDRA-11206 (Support large partitions on >>> the 3.0 sstable format), which reduces heap usage for other operations >>> (like repair, compactions) as well. >>> You can verify that by setting column_index_cache_size_in_kb in c.yaml >>> to a really high value like 10000000 - if you see the same behaviour in= 3.7 >>> with that setting, there=E2=80=99s not much you can do except upgrading= to 3.7 as >>> that change went into 3.6 and not into 3.0.x. >>> >>> =E2=80=94 >>> Robert Stupp >>> @snazy >>> >>> On 20 Jun 2016, at 18:13, Bhuvan Rawal wrote: >>> >>> Hi All, >>> >>> We are running Cassandra 3.0.3 on Production with Max Heap Size of 8GB. >>> There has been a consistent issue with nodetool repair for a while and >>> we have tried issuing it with multiple options --pr, --local as well, >>> sometimes node went down with Out of Memory error and at times nodes di= d >>> stopped connecting any connection, even jmx nodetool commands. >>> >>> On trying with same data on 3.7 Repair Ran successfully without >>> encountering any of the above mentioned issues. I then tried increasing >>> heap to 16GB on 3.0.3 and repair ran successfully. >>> >>> I then analyzed memory usage during nodetool repair for 3.0.3(16GB >>> heap) vs 3.7 (8GB Heap) and 3.0.3 occupied 11-14 GB at all times, >>> whereas 3.7 spiked between 1-4.5 GB while repair runs. As they ran on >>> same dataset and unrepaired data with full repair. >>> >>> We would like to know if it is a known bug that was fixed post 3.0.3 an= d >>> there could be a possible way by which we can run repair on 3.0.3 witho= ut >>> increasing heap size as for all other activities 8GB works for us. >>> >>> PFA the visualvm snapshots. >>> >>> >>> =E2=80=8B3.0.3 VisualVM Snapshot, consistent heap usage of greater than= 12 GB. >>> >>> >>> >>> =E2=80=8B3.7 VisualVM Snapshot, 8GB Max Heap and max heap usage till ab= out 5GB. >>> >>> Thanks & Regards, >>> Bhuvan Rawal >>> >>> >>> PS: In case if the snapshots are not visible, they can be viewed from >>> the following links: >>> 3.0.3: >>> https://s31.postimg.org/4e7ifsjaz/Screenshot_from_2016_06_20_21_06_09.p= ng >>> 3.7: >>> https://s31.postimg.org/xak32s9m3/Screenshot_from_2016_06_20_21_05_57.p= ng >>> >>> >>> >> > --047d7b8744f4ac03c60535dd9061 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
it could also be=C2=A0CASSANDRA-11412 if you have many sst= ables and vnodes

On Wed, Jun 22, 2016 at 2:50 PM, Bhuvan Rawal <bhu1rawal@gmail.com<= /a>> wrote:
Th= anks for the info Paulo, Robert. I tried further testing with other paramet= ers and it was prevalent. We could be either 11739, 11206. But im spektical= about 11739 because repair works well in 3.5 and 11739 seems to be fixed f= or 3.7/3.0.7.=C2=A0
We may possibly resolve this by increasing heap size thereby re= ducing some page cache bandwidth before upgrading to higher versions.
=

On Mon, Jun 20, 2016 at 10:00 PM, Paulo Motta <<= a href=3D"mailto:pauloricardomg@gmail.com" target=3D"_blank">pauloricardomg= @gmail.com> wrote:
You could also be hitting CASSANDRA-= 11739, which was fixed on 3.0.7 and could potentially cause OOMs for long-r= unning repairs.


2016-06-20 13:26 GMT-03:00 Robert Stupp <<= a href=3D"mailto:snazy@snazy.de" target=3D"_blank">snazy@snazy.de>:
One possibility might be CASSANDRA-11206 (Support large partitions on the= 3.0 sstable format), which reduces heap usage for other operations (like r= epair, compactions) as well.
You can verify that by setting=C2=A0column= _index_cache_size_in_kb in c.yaml to a really high value like=C2=A010000000= - if you see the same behaviour in 3.7 with that setting, there=E2=80=99s = not much you can do except upgrading to 3.7 as that change went into 3.6 an= d not into 3.0.x.

=E2=80=94
Robert Stupp
@snazy
<= /div>

On 20 Jun 2016, at 18:13, Bhu= van Rawal <bhu1= rawal@gmail.com> wrote:

= Hi All,

We are running Cassandra 3.0.3 on Production wit= h Max Heap Size of 8GB. There has been a consistent issue with nodetool repair for a while a= nd we have tried issuing it with multiple options --pr, --local as well, so= metimes node went down with Out of Memory error and at times nodes did stop= ped connecting any connection, even jmx nodetool commands.=C2=A0
=
On trying with same data on 3.7 Repair Ran successfully with= out encountering any of the above mentioned issues. I then tried increasing= heap to 16GB on 3.0.3 and repair ran successfully.

I then analyzed memory usage during nodetool repair for 3.0.3(16GB heap) vs 3.7 (8GB Heap) = and 3.0.3 occupied 11-14 = GB at all times, whereas 3.7 spiked between 1-4.5 GB while repair runs. As they ran on= same dataset and unrepaired data with full repair.=C2=A0

We would like to know if it is a known bug that was fixed post 3.0.= 3 and there could be a possible way by which we can run repair on 3.0.3 wit= hout increasing heap size as for all other activities 8GB works for us.

PFA the visualvm snapshots.

<Screenshot from 2016-06-20 21:06:09.png>=E2=80=8B3.0.3 VisualVM Snapshot, consistent heap usage of greater than 1= 2 GB.


<Screensh= ot from 2016-06-20 21:05:57.png>
=E2=80=8B3.7 VisualVM S= napshot, 8GB Max Heap and max heap usage till about 5GB.
=

Thanks & Regards,
Bhuvan Rawal
=

PS: In case if the snapshots are not visible,= they can be viewed from the following links:




--047d7b8744f4ac03c60535dd9061--