cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paulo Motta <pauloricard...@gmail.com>
Subject Re: High Heap Memory usage during nodetool repair in Cassandra 3.0.3
Date Mon, 20 Jun 2016 16:30:43 GMT
You could also be hitting CASSANDRA-11739, which was fixed on 3.0.7 and
could potentially cause OOMs for long-running repairs.

2016-06-20 13:26 GMT-03:00 Robert Stupp <snazy@snazy.de>:

> One possibility might be CASSANDRA-11206 (Support large partitions on the
> 3.0 sstable format), which reduces heap usage for other operations (like
> repair, compactions) as well.
> You can verify that by setting column_index_cache_size_in_kb in c.yaml to
> a really high value like 10000000 - if you see the same behaviour in 3.7
> with that setting, there’s not much you can do except upgrading to 3.7 as
> that change went into 3.6 and not into 3.0.x.
>
> —
> Robert Stupp
> @snazy
>
> On 20 Jun 2016, at 18:13, Bhuvan Rawal <bhu1rawal@gmail.com> wrote:
>
> Hi All,
>
> We are running Cassandra 3.0.3 on Production with Max Heap Size of 8GB.
> There has been a consistent issue with nodetool repair for a while and we
> have tried issuing it with multiple options --pr, --local as well,
> sometimes node went down with Out of Memory error and at times nodes did
> stopped connecting any connection, even jmx nodetool commands.
>
> On trying with same data on 3.7 Repair Ran successfully without
> encountering any of the above mentioned issues. I then tried increasing
> heap to 16GB on 3.0.3 and repair ran successfully.
>
> I then analyzed memory usage during nodetool repair for 3.0.3(16GB heap)
> vs 3.7 (8GB Heap) and 3.0.3 occupied 11-14 GB at all times, whereas 3.7
> spiked between 1-4.5 GB while repair runs. As they ran on same dataset
> and unrepaired data with full repair.
>
> We would like to know if it is a known bug that was fixed post 3.0.3 and
> there could be a possible way by which we can run repair on 3.0.3 without
> increasing heap size as for all other activities 8GB works for us.
>
> PFA the visualvm snapshots.
>
> <Screenshot from 2016-06-20 21:06:09.png>
> ​3.0.3 VisualVM Snapshot, consistent heap usage of greater than 12 GB.
>
>
> <Screenshot from 2016-06-20 21:05:57.png>
> ​3.7 VisualVM Snapshot, 8GB Max Heap and max heap usage till about 5GB.
>
> Thanks & Regards,
> Bhuvan Rawal
>
>
> PS: In case if the snapshots are not visible, they can be viewed from the
> following links:
> 3.0.3:
> https://s31.postimg.org/4e7ifsjaz/Screenshot_from_2016_06_20_21_06_09.png
> 3.7:
> https://s31.postimg.org/xak32s9m3/Screenshot_from_2016_06_20_21_05_57.png
>
>
>

Mime
View raw message