cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitan Kainth <ni...@bamlabs.com>
Subject Re: Long running compaction on huge hint table.
Date Tue, 16 May 2017 14:26:55 GMT
You can control compaction with nodetool compactionthroughput but it will just slow down compaction
and give resources for application, however it's not a fix.

Sent from my iPhone

> On May 16, 2017, at 9:15 AM, varun saluja <saluja50@gmail.com> wrote:
> 
> Thanks Nitan.
> Appreciate your help.
> 
> Can anyone suggest parameter change or something which can help in this situation.
> 
> Regards,
> Varun 
> 
> Sent from my iPhone
> 
>> On 16-May-2017, at 7:31 PM, Nitan Kainth <nitan@bamlabs.com> wrote:
>> 
>> If target table is dropped then you can remove its hints but there could be more
hints from other table. If it has tables of your interest , then I won't comment on truncating
hints.
>> 
>> Size of hints depends on Kafka load , looks like you had overloaded the cluster during
data load and not hints are just recovering from it. I would say wait until cluster comes
to normal state. May be some other expert can suggest an alternate.
>> 
>> Sent from my iPhone
>> 
>>> On May 16, 2017, at 8:50 AM, varun saluja <saluja50@gmail.com> wrote:
>>> 
>>> Thanks for update.
>>> I could see lot of io waits. This causing  Gc and mutation drops .
>>> But as i mentioned we do not have high load for now. Hint replays are creating
such high disk I/O.
>>> compactionstats show very high hint bytes like 780gb around. Is this normal?
>>> 
>>> Just mentioning we are using flash disks.
>>> 
>>> In such case, if i run truncatehints , will it remove or decrease size of hints
bytes in compaction stats. I can trigger repair therafter.
>>> Please let me know if any recommendation on same.
>>> 
>>> Also , table which we dumped from kafka which created this much hints and compaction
pendings is also dropped today. Because we have to redump table again once cluster is stable.
>>> 
>>> Regards,
>>> Varun
>>> 
>>> Sent from my iPhone
>>> 
>>>> On 16-May-2017, at 6:59 PM, Nitan Kainth <nitan@bamlabs.com> wrote:
>>>> 
>>>> Yes but it means data has to be replicated using repair.
>>>> 
>>>> Hints are out come of unhealthy nodes, focus on finding why you have mutation
drops, is it node, io or network etc. ideally you shouldn't see increasing hints all the time.
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>> On May 16, 2017, at 7:58 AM, varun saluja <saluja50@gmail.com>
wrote:
>>>>> 
>>>>> Hi Nitan,
>>>>> 
>>>>> Thanks for response.
>>>>> 
>>>>> Yes, I could see mutation drops and increase count in system.hints. Is
there any way , i can proceed to truncate hints like using nodetool truncatehints.
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> Varun Saluja
>>>>> 
>>>>>> On 16 May 2017 at 17:52, Nitan Kainth <nitan@bamlabs.com> wrote:
>>>>>> Do you see mutation drops?
>>>>>> Select count from system.hints; is it increasing?
>>>>>> 
>>>>>> Sent from my iPhone
>>>>>> 
>>>>>>> On May 16, 2017, at 5:52 AM, varun saluja <saluja50@gmail.com>
wrote:
>>>>>>> 
>>>>>>> Hi Experts,
>>>>>>> 
>>>>>>> We are facing issue on production cluster. Compaction on system.hint
table is running from last 2 days.
>>>>>>> 
>>>>>>> 
>>>>>>> pending tasks: 1
>>>>>>>    compaction type   keyspace   table     completed         
total                      unit   progress
>>>>>>>               Compaction     system   hints   20623021829   877874092407
  bytes      2.35%
>>>>>>> Active compaction remaining time :   0h27m15s
>>>>>>> 
>>>>>>> 
>>>>>>> Active compaction remaining time shows in minutes.  But, this
is job is running like indefinitely.
>>>>>>> 
>>>>>>> We have 3 node cluster V 2.1.7. And we ran  write intensive job
last week on particular table.
>>>>>>> Compaction on this table finished but hint table size is growing
continuously.
>>>>>>> 
>>>>>>> Can someone Please help me.
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks & Regards,
>>>>>>> Varun Saluja
>>>>>>> 
>>>>> 

Mime
View raw message