cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick McFadin <pmcfa...@gmail.com>
Subject Re: A few misbehaving nodes
Date Wed, 20 Apr 2016 18:47:23 GMT
Can you show the output of a tpstats on one of the effected nodes? That
will give some indication where the trouble might be.

Patrick

On Tue, Apr 19, 2016 at 6:54 AM, sai krishnam raju potturi <
pskraju88@gmail.com> wrote:

> hi;
>    do we see any hung process like Repairs on those 3 nodes?  what does
> "nodetool netstats" show??
>
> thanks
> Sai
>
> On Tue, Apr 19, 2016 at 8:24 AM, Erik Forsberg <forsberg@opera.com> wrote:
>
>> Hi!
>>
>> I have this problem where 3 of my 84 nodes misbehave with too long GC
>> times, leading to them being marked as DN.
>>
>> This happens when I load data to them using CQL from a hadoop job, so
>> quite a lot of inserts at a time. The CQL loading job is using
>> TokenAwarePolicy with fallback to DCAwareRoundRobinPolicy. Cassandra java
>> driver version 2.1.7.1 is in use.
>>
>> My other observation is that around the time the GC starts to work like
>> crazy, there is a lot of outbound network traffic from the troublesome
>> nodes. If a healthy node has around 25 Mbit/s in, 25 Mbit/s out, an
>> unhealthy sees 25 Mbit/s in, 200 Mbit/s out.
>>
>> So, something is iffy with these 3 nodes, but I have some trouble finding
>> out exactly what makes them differ.
>>
>> This is Cassandra 2.0.13 (yes, old) using vnodes. Keyspace is using
>> NetworkTopologyStrategy with replication 2, in one datacenter.
>>
>> One thing I know I'm doing wrong is that I have slightly differing number
>> of hosts in each of my 6 chassies (One of them have 15 nodes, one of have
>> 13, the remaining have 14). Could what I'm seeing here be the effect of
>> that?
>>
>> Other ideas on what could be wrong? Some kind of vnode imbalance? How can
>> I diagnose that? What metrics should I be looking at?
>>
>> Thanks,
>> \EF
>>
>>
>>
>

Mime
View raw message