cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Andersen <rich...@andersenfamily.us>
Subject Re: How to read content of hints file and apply them manually?
Date Tue, 28 Jan 2020 19:05:42 GMT
I am in agreement with Patrick, this is a typical symptom of saturated IO. Are there a high
of drops and/or pending compactions?

Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: Patrick McFadin <pmcfadin@gmail.com>
Sent: Tuesday, January 28, 2020 11:25:49 AM
To: user@cassandra.apache.org <user@cassandra.apache.org>
Subject: Re: How to read content of hints file and apply them manually?

Just to add in here. Any time I see any hints on a cluster, that's like seeing smoke. If you
can't explain it, you have a fire somewhere and it's not going to get any better.

By the few messages I've seen, I would start by looking at your IO subsystem on your nodes.
Do you have enough throughput to write and read at the same time? These are exactly the symptoms
I see when running Cassandra on a SAN or NAS.

Patrick

On Mon, Jan 27, 2020 at 8:17 PM Surbhi Gupta <surbhi.gupta01@gmail.com<mailto:surbhi.gupta01@gmail.com>>
wrote:
We tried to tune sethintedhandoffthrottlekb to 100 , 1024 , 10240 but nothing helped .
Our hints related parameters are as below, if you don't find any parameter below then it is
not set in our environment and should be of the default value.

max_hint_window_in_ms: 10800000 # 3 hours

hinted_handoff_enabled: true

hinted_handoff_throttle_in_kb: 100

max_hints_delivery_threads: 8

hints_directory: /var/lib/cassandra/hints

hints_flush_period_in_ms: 10000

max_hints_file_size_in_mb: 128

On Mon, 27 Jan 2020 at 18:34, Jeff Jirsa <jjirsa@gmail.com<mailto:jjirsa@gmail.com>>
wrote:

The high cpu is probably the hints getting replayed slamming the write path

Slowing it down with the hint throttle may help

It’s not instant.

On Jan 27, 2020, at 6:05 PM, Erick Ramirez <flightctlr@gmail.com<mailto:flightctlr@gmail.com>>
wrote:


Increase the max_hint_window_in_ms setting in cassandra.yaml to more than 3 hours, perhaps
6 hours. If the issue still persists networking may need to be tested for bandwidth issues.

Just a note of warning about bumping up the hint window without understanding the pros and
cons. Be aware that doubling it means:

  *   you'll end up doubling the size of stored hints in the hints_directory
  *   there'll be twice as much hints to replay when node(s) come back online

There's always 2 sides to fiddling with the knobs in C*. Cheers!
Mime
View raw message