cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ezra Stuetzel <ezra.stuet...@riskiq.net>
Subject Re: large system hint partition
Date Mon, 19 Sep 2016 03:43:09 GMT
Yeah I tried that, but oddly the table had nothing in it.

I changed the compaction strategy from leveled to sizetierd and ran a major
compaction on each node. I haven't seen the message logged on any node in a
few days which makes me think that fixed it because it is normally logged
multiple times per day.

On Sun, Sep 18, 2016 at 4:29 AM, Carlos Alonso <info@mrcalonso.com> wrote:

> By inspecting the contents on your system.hints table, specifically the
> host_id column, you can see which is the destination host of those hints
> and check if it is one of the alive or dead ones.
>
> Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
>
> On 18 September 2016 at 04:35, Ezra Stuetzel <ezra.stuetzel@riskiq.net>
> wrote:
>
>> Hey Nicolas,
>>
>> There are no dead nodes. 'nodetool status' and 'nodetool describecluster'
>> both show 4 healthy nodes. In the past we have had some nodes we eliminated
>> by using 'nodetool assassinate'. However, I checked system.peers table on
>> all 4 of our nodes and they each show 3 peers as expected. So it doesn't
>> appear that any nodes have any awareness of an unreachable node which could
>> be causing hints to back up. Any ideas for further troubleshooting what the
>> hints are?
>>
>> Thanks,
>> Ezra
>>
>> On Fri, Sep 16, 2016 at 4:13 PM, Nicolas Douillet <
>> nicolas.douillet@gmail.com> wrote:
>>
>>> Hi Erza,
>>>
>>> Have you a dead node in your cluster?
>>> Because the coordinator stores a hint about dead replicas in the local
>>> system.hints when a node is dead or didn't respond to a write request.
>>>
>>> --
>>> Nicolas
>>>
>>>
>>>
>>> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel <ezra.stuetzel@riskiq.net>
>>> a écrit :
>>>
>>>> What would be the likely causes of large system hint partitions?
>>>> Normally large partition warnings are for user defined tables which they
>>>> are writing large partitions to. In this case, it appears C* is writing
>>>> large partitions to the system.hints table. Gossip is not backed up.
>>>>
>>>> version: C* 2.2.7
>>>>
>>>> WARN  [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
>>>> BigTableWriter.java:184 - Writing large partition
>>>> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Ezra
>>>>
>>>
>>
>

Mime
View raw message