cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Terje Marthinussen <tmarthinus...@gmail.com>
Subject Re: 0.8 loosing nodes?
Date Mon, 25 Apr 2011 16:51:11 GMT
Got just enough time to look at this done today to verify that:

Sometimes nodes (under pressure) fails to send heartbeats for  long
enough to get marked as dead by other nodes (why is a good question,
which I need to check better. Does not seem to be GC).

The node does however start sending heartbeats again and other nodes
log that they receive the heartbeats,  but this will not get it marked
as UP again until restarted.

So, seems like 2 issues:
- Nodes pausing (may be just node overload)
- Nodes are not marked as UP unless restarted

Regards,
Terje

On 24 Apr 2011, at 23:24, Terje Marthinussen <tmarthinussen@gmail.com> wrote:

> World as seen from .81 in the below ring
> .81     Up     Normal  85.55 GB        8.33%   Token(bytes[30])
> .82     Down   Normal  83.23 GB        8.33%   Token(bytes[313230])
> .83     Up     Normal  70.43 GB        8.33%   Token(bytes[313437])
> .84     Up     Normal  81.7 GB         8.33%   Token(bytes[313836])
> .85     Up     Normal  108.39 GB       8.33%   Token(bytes[323336])
> .86     Up     Normal  126.19 GB       8.33%   Token(bytes[333234])
> .87     Up     Normal  127.16 GB       8.33%   Token(bytes[333939])
> .88     Up     Normal  135.92 GB       8.33%   Token(bytes[343739])
> .89     Up     Normal  117.1 GB        8.33%   Token(bytes[353730])
> .90     Up     Normal  101.67 GB       8.33%   Token(bytes[363635])
> .91     Down   Normal  88.33 GB        8.33%   Token(bytes[383036])
> .92     Up     Normal  129.95 GB       8.33%   Token(bytes[6a])
>
>
> From .82
> .81     Down   Normal  85.55 GB        8.33%   Token(bytes[30])
> .82     Up     Normal  83.23 GB        8.33%   Token(bytes[313230])
> .83     Up     Normal  70.43 GB        8.33%   Token(bytes[313437])
> .84     Up     Normal  81.7 GB         8.33%   Token(bytes[313836])
> .85     Up     Normal  108.39 GB       8.33%   Token(bytes[323336])
> .86     Up     Normal  126.19 GB       8.33%   Token(bytes[333234])
> .87     Up     Normal  127.16 GB       8.33%   Token(bytes[333939])
> .88     Up     Normal  135.92 GB       8.33%   Token(bytes[343739])
> .89     Up     Normal  117.1 GB        8.33%   Token(bytes[353730])
> .90     Up     Normal  101.67 GB       8.33%   Token(bytes[363635])
> .91     Down   Normal  88.33 GB        8.33%   Token(bytes[383036])
> .92     Up     Normal  129.95 GB       8.33%   Token(bytes[6a])
>
> From .84
> 10.10.42.81     Down   Normal  85.55 GB        8.33%   Token(bytes[30])
> 10.10.42.82     Down   Normal  83.23 GB        8.33%   Token(bytes[313230])
> 10.10.42.83     Up     Normal  70.43 GB        8.33%   Token(bytes[313437])
> 10.10.42.84     Up     Normal  81.7 GB         8.33%   Token(bytes[313836])
> 10.10.42.85     Up     Normal  108.39 GB       8.33%   Token(bytes[323336])
> 10.10.42.86     Up     Normal  126.19 GB       8.33%   Token(bytes[333234])
> 10.10.42.87     Up     Normal  127.16 GB       8.33%   Token(bytes[333939])
> 10.10.42.88     Up     Normal  135.92 GB       8.33%   Token(bytes[343739])
> 10.10.42.89     Up     Normal  117.1 GB        8.33%   Token(bytes[353730])
> 10.10.42.90     Up     Normal  101.67 GB       8.33%   Token(bytes[363635])
> 10.10.42.91     Down   Normal  88.33 GB        8.33%   Token(bytes[383036])
> 10.10.42.92     Up     Normal  129.95 GB       8.33%   Token(bytes[6a])
>
> All of the nodes seems to be working when looked at individually and I can see on for
instance .84 that
>  INFO [ScheduledTasks:1] 2011-04-24 04:51:53,164 Gossiper.java (line 611) InetAddress
/.81 is now dead.
>
> but there is no other messages related to the nodes "dissappearing"  as far as I can
see in the 18 hours since that message occured.
>
> Restarting seems to recover things, but nodes seems to go away again (0.8 also seem to
be prone to commit logs being unreadable in some cases?)
>
> This is 0.8 build from trunk last Friday.
>
> I will try to enable some more debugging tomorrow to see if there is something interesting,
just curious if anyone else had noticed something like this.
>
> Regards,
> Terje
>
>

Mime
View raw message