cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Atul Saroha <atul.sar...@snapdeal.com>
Subject Re: High Heap Memory usage during nodetool repair in Cassandra 3.0.3
Date Mon, 20 Jun 2016 17:51:48 GMT
We have tried this with 3.5 and there also heap usage was optimized as in
3.7.
Though we have to roll-back from 3.5 to 3.0.3 due to CASSANDRA-11513
<https://issues.apache.org/jira/browse/CASSANDRA-11513>.

---------------------------------------------------------------------------------------------------------------------
Atul Saroha
*Lead Software Engineer*
*M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
Plot # 362, ASF Centre - Tower A, Udyog Vihar,
 Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA

On Mon, Jun 20, 2016 at 10:00 PM, Paulo Motta <pauloricardomg@gmail.com>
wrote:

> You could also be hitting CASSANDRA-11739, which was fixed on 3.0.7 and
> could potentially cause OOMs for long-running repairs.
>
>
> 2016-06-20 13:26 GMT-03:00 Robert Stupp <snazy@snazy.de>:
>
>> One possibility might be CASSANDRA-11206 (Support large partitions on the
>> 3.0 sstable format), which reduces heap usage for other operations (like
>> repair, compactions) as well.
>> You can verify that by setting column_index_cache_size_in_kb in c.yaml to
>> a really high value like 10000000 - if you see the same behaviour in 3.7
>> with that setting, there’s not much you can do except upgrading to 3.7 as
>> that change went into 3.6 and not into 3.0.x.
>>
>> —
>> Robert Stupp
>> @snazy
>>
>> On 20 Jun 2016, at 18:13, Bhuvan Rawal <bhu1rawal@gmail.com> wrote:
>>
>> Hi All,
>>
>> We are running Cassandra 3.0.3 on Production with Max Heap Size of 8GB.
>> There has been a consistent issue with nodetool repair for a while and
>> we have tried issuing it with multiple options --pr, --local as well,
>> sometimes node went down with Out of Memory error and at times nodes did
>> stopped connecting any connection, even jmx nodetool commands.
>>
>> On trying with same data on 3.7 Repair Ran successfully without
>> encountering any of the above mentioned issues. I then tried increasing
>> heap to 16GB on 3.0.3 and repair ran successfully.
>>
>> I then analyzed memory usage during nodetool repair for 3.0.3(16GB heap)
>> vs 3.7 (8GB Heap) and 3.0.3 occupied 11-14 GB at all times, whereas 3.7
>> spiked between 1-4.5 GB while repair runs. As they ran on same dataset
>> and unrepaired data with full repair.
>>
>> We would like to know if it is a known bug that was fixed post 3.0.3 and
>> there could be a possible way by which we can run repair on 3.0.3 without
>> increasing heap size as for all other activities 8GB works for us.
>>
>> PFA the visualvm snapshots.
>>
>> <Screenshot from 2016-06-20 21:06:09.png>
>> ​3.0.3 VisualVM Snapshot, consistent heap usage of greater than 12 GB.
>>
>>
>> <Screenshot from 2016-06-20 21:05:57.png>
>> ​3.7 VisualVM Snapshot, 8GB Max Heap and max heap usage till about 5GB.
>>
>> Thanks & Regards,
>> Bhuvan Rawal
>>
>>
>> PS: In case if the snapshots are not visible, they can be viewed from the
>> following links:
>> 3.0.3:
>> https://s31.postimg.org/4e7ifsjaz/Screenshot_from_2016_06_20_21_06_09.png
>> 3.7:
>> https://s31.postimg.org/xak32s9m3/Screenshot_from_2016_06_20_21_05_57.png
>>
>>
>>
>

Mime
View raw message