cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From varun saluja <saluj...@gmail.com>
Subject Re: Long running compaction on huge hint table.
Date Tue, 16 May 2017 13:23:02 GMT
Hi,

Could see intermittent GCs and mutation drops.

*System log reports:*

INFO  [Service Thread]  GCInspector.java:252 - ParNew GC in 3816ms.  CMS
Old Gen: 4663180720 -> 5520012520; Par Eden Space: 1718091776 -> 0; Par
Survivor Space: 0 -> 214695936
INFO  [ScheduledTasks:1] MessagingService.java:888 - 228 MUTATION messages
dropped in last 5000ms

PS: As of now , there is no significant load on our cluster. The only load
is of these hints been replayed.

Can you Please help.

Regards,
Varun Saluja

On 16 May 2017 at 18:28, varun saluja <saluja50@gmail.com> wrote:

> Hi Nitan,
>
> Thanks for response.
>
> Yes, I could see mutation drops and increase count in system.hints. Is
> there any way , i can proceed to truncate hints like using nodetool
> truncatehints.
>
>
> Regards,
> Varun Saluja
>
> On 16 May 2017 at 17:52, Nitan Kainth <nitan@bamlabs.com> wrote:
>
>> Do you see mutation drops?
>> Select count from system.hints; is it increasing?
>>
>> Sent from my iPhone
>>
>> On May 16, 2017, at 5:52 AM, varun saluja <saluja50@gmail.com> wrote:
>>
>> Hi Experts,
>>
>> We are facing issue on production cluster. Compaction on system.hint
>> table is running from last 2 days.
>>
>>
>> pending tasks: 1
>>    compaction type   keyspace   table     completed          total
>>               unit   progress
>>               Compaction     system   hints   20623021829
>> *877874092407*   bytes      2.35%
>> Active compaction remaining time :   0h27m15s
>>
>>
>> Active compaction remaining time shows in minutes.  But, this is job is
>> running like indefinitely.
>>
>> We have 3 node cluster V 2.1.7. And we ran  write intensive job last week
>> on particular table.
>> Compaction on this table finished but hint table size is growing
>> continuously.
>>
>> Can someone Please help me.
>>
>>
>> Thanks & Regards,
>> Varun Saluja
>>
>>
>

Mime
View raw message