incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Talbot <btal...@aeriagames.com>
Subject Re: High disk read throughput on only one node.
Date Wed, 19 Dec 2012 23:36:39 GMT
Oh, you're on ec2.  Maybe the dynamic snitch is detecting that one node is
performing better than the others so is routing more traffic to it?

http://www.datastax.com/docs/1.1/configuration/node_configuration#dynamic-snitch-badness-threshold

-Bryan




On Wed, Dec 19, 2012 at 2:30 PM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:

> @Aaron
> "Is there a sustained difference or did it settle back ? "
>
> Sustained, clearly. During the day all nodes read at about 6MB/s while
> this one reads at 30-40 MB/s. At night while other reads 2MB/s the "broken"
> nodes reads at 8-10MB/s
>
> "Could this have been compaction or repair or upgrade tables working ? "
>
> Was my first thought but definitely no. this occurs continuously.
>
> "Do the read / write counts available in nodetool cfstats show anything
> different ? "
>
> The cfstats shows different counts (a lot less reads/writes for the "bad"
> node)  but they didn't join the ring at the same time. I join you the
> cfstats just in case it could help somehow.
>
> Node  38: http://pastebin.com/ViS1MR8d (bad one)
> Node  32: http://pastebin.com/MrSTHH9F
> Node 154: http://pastebin.com/7p0Usvwd
>
> @Bryan
>
>  "clients always connect to that server"
>
> I didn't join it in the screenshot from AWS console, but AWS report an
> (almost) equal network within the nodes (same for output and cpu). The cpu
> load is a lot higher in the broken node as shown by the OpsCenter, but
> that's due to the high iowait...)
>



-- 
Bryan Talbot
Architect / Platform team lead, Aeria Games and Entertainment
Silicon Valley | Berlin | Tokyo | Sao Paulo

Mime
View raw message