cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ZAIDI, ASAD A" <az1...@att.com>
Subject RE: MUTATION messages were dropped in last 5000 ms for cross node timeout
Date Fri, 21 Jul 2017 14:46:25 GMT
Hi Akhil,

Thank you for your reply. Previously, I did ‘tune’ various timeouts – basically increased
them a bit but none of those parameter listed in the link matches with that “were dropped
in last 5000 ms”.
I was wondering from where that [5000ms] number is coming from when,  like I mentioned before,
none of any timeout parameter settings matches that #!

Load is intermittently high but again cpu queue length never goes beyond medium depth. I wonder
if there is some internal limit that I’m still not aware of.

Thanks/Asad


From: Akhil Mehra [mailto:akhilmehra@gmail.com]
Sent: Thursday, July 20, 2017 3:47 PM
To: user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

Hi Asad,

http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped<https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org_doc_latest_faq_index.html-23why-2Dmessage-2Ddropped&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=WcHuHKcjg2YCsAbw2NR_0-CiHr9JNxtCzYikia16mpo&s=0_0pQfoOZLuswpQ_lE-AU2bTMFLgRbR4k4Kh8vEOZSk&e=>

As mentioned in the link above this is a load shedding mechanism used by Cassandra.

Is you cluster under heavy load?

Regards,
Akhil


On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <az192g@att.com<mailto:az192g@att.com>>
wrote:

Hello Folks –

I’m using apache-cassandra 2.2.8.

I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout:
true] is set and NTP server is also running correcting clock drift on 16node cluster. I do
not see pending or blocked HintedHandoff  in tpstats output though there are bunch of MUTATIONS
dropped observed.

<start timeout message >
INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages
were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout
<end timeout message >

I’m seeking help here if you please let me know what I need to check in order to address
these cross node timeouts.

Thank you,
Asad

Mime
View raw message