It looks like nobody has already experiment this kind of trouble or even has a clue about it.

Under heavy load this creates a high latency (because of iowait) in my app in prod and we can't handle it longer. If there is nothing new in the few upcoming days I think I'll drop this node and replace it, hopping this will fix my issue...

I wait a bit more because I am hopping we will find out what is the issue and this will help the C* community.

2012/12/20 Alain RODRIGUEZ <>

"routing more traffic to it?"

So shouldn't I see more "network in" on that node in the AWS console ?

It seems that each node is recieving and sending an equal amount of data.

What value should I use for dynamic-snitch-badness-threshold to give it a try ?

Le 20 déc. 2012 00:37, "Bryan Talbot" <> a écrit :

Oh, you're on ec2.  Maybe the dynamic snitch is detecting that one node is performing better than the others so is routing more traffic to it?


On Wed, Dec 19, 2012 at 2:30 PM, Alain RODRIGUEZ <> wrote:
"Is there a sustained difference or did it settle back ? "

Sustained, clearly. During the day all nodes read at about 6MB/s while this one reads at 30-40 MB/s. At night while other reads 2MB/s the "broken" nodes reads at 8-10MB/s

"Could this have been compaction or repair or upgrade tables working ? "

Was my first thought but definitely no. this occurs continuously.

"Do the read / write counts available in nodetool cfstats show anything different ? "

The cfstats shows different counts (a lot less reads/writes for the "bad" node)  but they didn't join the ring at the same time. I join you the cfstats just in case it could help somehow.

Node  38: (bad one)


"clients always connect to that server"

I didn't join it in the screenshot from AWS console, but AWS report an (almost) equal network within the nodes (same for output and cpu). The cpu load is a lot higher in the broken node as shown by the OpsCenter, but that's due to the high iowait...)

Bryan Talbot
Architect / Platform team lead, Aeria Games and Entertainment
Silicon Valley | Berlin | Tokyo | Sao Paulo