cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ZAIDI, ASAD A" <>
Subject RE: MUTATION messages were dropped in last 5000 ms for cross node timeout
Date Fri, 21 Jul 2017 14:46:25 GMT
Hi Akhil,

Thank you for your reply. Previously, I did ‘tune’ various timeouts – basically increased
them a bit but none of those parameter listed in the link matches with that “were dropped
in last 5000 ms”.
I was wondering from where that [5000ms] number is coming from when,  like I mentioned before,
none of any timeout parameter settings matches that #!

Load is intermittently high but again cpu queue length never goes beyond medium depth. I wonder
if there is some internal limit that I’m still not aware of.


From: Akhil Mehra []
Sent: Thursday, July 20, 2017 3:47 PM
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

Hi Asad,<>

As mentioned in the link above this is a load shedding mechanism used by Cassandra.

Is you cluster under heavy load?


On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <<>>

Hello Folks –

I’m using apache-cassandra 2.2.8.

I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout:
true] is set and NTP server is also running correcting clock drift on 16node cluster. I do
not see pending or blocked HintedHandoff  in tpstats output though there are bunch of MUTATIONS
dropped observed.

<start timeout message >
INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 - MUTATION messages
were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
<end timeout message >

I’m seeking help here if you please let me know what I need to check in order to address
these cross node timeouts.

Thank you,

View raw message