cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Eriksson <krum...@gmail.com>
Subject Re: High Heap Memory usage during nodetool repair in Cassandra 3.0.3
Date Wed, 22 Jun 2016 13:03:34 GMT
it could also be CASSANDRA-11412 if you have many sstables and vnodes

On Wed, Jun 22, 2016 at 2:50 PM, Bhuvan Rawal <bhu1rawal@gmail.com> wrote:

> Thanks for the info Paulo, Robert. I tried further testing with other
> parameters and it was prevalent. We could be either 11739, 11206. But im
> spektical about 11739 because repair works well in 3.5 and 11739 seems to
> be fixed for 3.7/3.0.7.
>
> We may possibly resolve this by increasing heap size thereby reducing some
> page cache bandwidth before upgrading to higher versions.
>
> On Mon, Jun 20, 2016 at 10:00 PM, Paulo Motta <pauloricardomg@gmail.com>
> wrote:
>
>> You could also be hitting CASSANDRA-11739, which was fixed on 3.0.7 and
>> could potentially cause OOMs for long-running repairs.
>>
>>
>> 2016-06-20 13:26 GMT-03:00 Robert Stupp <snazy@snazy.de>:
>>
>>> One possibility might be CASSANDRA-11206 (Support large partitions on
>>> the 3.0 sstable format), which reduces heap usage for other operations
>>> (like repair, compactions) as well.
>>> You can verify that by setting column_index_cache_size_in_kb in c.yaml
>>> to a really high value like 10000000 - if you see the same behaviour in 3.7
>>> with that setting, there’s not much you can do except upgrading to 3.7 as
>>> that change went into 3.6 and not into 3.0.x.
>>>
>>> —
>>> Robert Stupp
>>> @snazy
>>>
>>> On 20 Jun 2016, at 18:13, Bhuvan Rawal <bhu1rawal@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> We are running Cassandra 3.0.3 on Production with Max Heap Size of 8GB.
>>> There has been a consistent issue with nodetool repair for a while and
>>> we have tried issuing it with multiple options --pr, --local as well,
>>> sometimes node went down with Out of Memory error and at times nodes did
>>> stopped connecting any connection, even jmx nodetool commands.
>>>
>>> On trying with same data on 3.7 Repair Ran successfully without
>>> encountering any of the above mentioned issues. I then tried increasing
>>> heap to 16GB on 3.0.3 and repair ran successfully.
>>>
>>> I then analyzed memory usage during nodetool repair for 3.0.3(16GB
>>> heap) vs 3.7 (8GB Heap) and 3.0.3 occupied 11-14 GB at all times,
>>> whereas 3.7 spiked between 1-4.5 GB while repair runs. As they ran on
>>> same dataset and unrepaired data with full repair.
>>>
>>> We would like to know if it is a known bug that was fixed post 3.0.3 and
>>> there could be a possible way by which we can run repair on 3.0.3 without
>>> increasing heap size as for all other activities 8GB works for us.
>>>
>>> PFA the visualvm snapshots.
>>>
>>> <Screenshot from 2016-06-20 21:06:09.png>
>>> ​3.0.3 VisualVM Snapshot, consistent heap usage of greater than 12 GB.
>>>
>>>
>>> <Screenshot from 2016-06-20 21:05:57.png>
>>> ​3.7 VisualVM Snapshot, 8GB Max Heap and max heap usage till about 5GB.
>>>
>>> Thanks & Regards,
>>> Bhuvan Rawal
>>>
>>>
>>> PS: In case if the snapshots are not visible, they can be viewed from
>>> the following links:
>>> 3.0.3:
>>> https://s31.postimg.org/4e7ifsjaz/Screenshot_from_2016_06_20_21_06_09.png
>>> 3.7:
>>> https://s31.postimg.org/xak32s9m3/Screenshot_from_2016_06_20_21_05_57.png
>>>
>>>
>>>
>>
>

Mime
View raw message