incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: 0.8 loosing nodes?
Date Mon, 25 Apr 2011 17:21:00 GMT
I bet the problem is with the other tasks on the executor that Gossip
heartbeat runs on.

I see at least two that could cause blocking: hint cleanup
post-delivery and flush-expired-memtables, both of which call
forceFlush which will block if the flush queue + threads are full.

We've run into this before (CASSANDRA-2253); we should move Gossip
back to its own dedicated executor or it will keep happening whenever
someone accidentally puts something on the "shared" executor that can
block.

Created https://issues.apache.org/jira/browse/CASSANDRA-2554 to fix
this.  Thanks for tracking down the problem!

On Mon, Apr 25, 2011 at 11:51 AM, Terje Marthinussen
<tmarthinussen@gmail.com> wrote:
> Got just enough time to look at this done today to verify that:
>
> Sometimes nodes (under pressure) fails to send heartbeats for  long
> enough to get marked as dead by other nodes (why is a good question,
> which I need to check better. Does not seem to be GC).
>
> The node does however start sending heartbeats again and other nodes
> log that they receive the heartbeats,  but this will not get it marked
> as UP again until restarted.
>
> So, seems like 2 issues:
> - Nodes pausing (may be just node overload)
> - Nodes are not marked as UP unless restarted
>
> Regards,
> Terje
>
> On 24 Apr 2011, at 23:24, Terje Marthinussen <tmarthinussen@gmail.com> wrote:
>
>> World as seen from .81 in the below ring
>> .81     Up     Normal  85.55 GB        8.33%   Token(bytes[30])
>> .82     Down   Normal  83.23 GB        8.33%   Token(bytes[313230])
>> .83     Up     Normal  70.43 GB        8.33%   Token(bytes[313437])
>> .84     Up     Normal  81.7 GB         8.33%   Token(bytes[313836])
>> .85     Up     Normal  108.39 GB       8.33%   Token(bytes[323336])
>> .86     Up     Normal  126.19 GB       8.33%   Token(bytes[333234])
>> .87     Up     Normal  127.16 GB       8.33%   Token(bytes[333939])
>> .88     Up     Normal  135.92 GB       8.33%   Token(bytes[343739])
>> .89     Up     Normal  117.1 GB        8.33%   Token(bytes[353730])
>> .90     Up     Normal  101.67 GB       8.33%   Token(bytes[363635])
>> .91     Down   Normal  88.33 GB        8.33%   Token(bytes[383036])
>> .92     Up     Normal  129.95 GB       8.33%   Token(bytes[6a])
>>
>>
>> From .82
>> .81     Down   Normal  85.55 GB        8.33%   Token(bytes[30])
>> .82     Up     Normal  83.23 GB        8.33%   Token(bytes[313230])
>> .83     Up     Normal  70.43 GB        8.33%   Token(bytes[313437])
>> .84     Up     Normal  81.7 GB         8.33%   Token(bytes[313836])
>> .85     Up     Normal  108.39 GB       8.33%   Token(bytes[323336])
>> .86     Up     Normal  126.19 GB       8.33%   Token(bytes[333234])
>> .87     Up     Normal  127.16 GB       8.33%   Token(bytes[333939])
>> .88     Up     Normal  135.92 GB       8.33%   Token(bytes[343739])
>> .89     Up     Normal  117.1 GB        8.33%   Token(bytes[353730])
>> .90     Up     Normal  101.67 GB       8.33%   Token(bytes[363635])
>> .91     Down   Normal  88.33 GB        8.33%   Token(bytes[383036])
>> .92     Up     Normal  129.95 GB       8.33%   Token(bytes[6a])
>>
>> From .84
>> 10.10.42.81     Down   Normal  85.55 GB        8.33%   Token(bytes[30])
>> 10.10.42.82     Down   Normal  83.23 GB        8.33%   Token(bytes[313230])
>> 10.10.42.83     Up     Normal  70.43 GB        8.33%   Token(bytes[313437])
>> 10.10.42.84     Up     Normal  81.7 GB         8.33%   Token(bytes[313836])
>> 10.10.42.85     Up     Normal  108.39 GB       8.33%   Token(bytes[323336])
>> 10.10.42.86     Up     Normal  126.19 GB       8.33%   Token(bytes[333234])
>> 10.10.42.87     Up     Normal  127.16 GB       8.33%   Token(bytes[333939])
>> 10.10.42.88     Up     Normal  135.92 GB       8.33%   Token(bytes[343739])
>> 10.10.42.89     Up     Normal  117.1 GB        8.33%   Token(bytes[353730])
>> 10.10.42.90     Up     Normal  101.67 GB       8.33%   Token(bytes[363635])
>> 10.10.42.91     Down   Normal  88.33 GB        8.33%   Token(bytes[383036])
>> 10.10.42.92     Up     Normal  129.95 GB       8.33%   Token(bytes[6a])
>>
>> All of the nodes seems to be working when looked at individually and I can see on
for instance .84 that
>>  INFO [ScheduledTasks:1] 2011-04-24 04:51:53,164 Gossiper.java (line 611) InetAddress
/.81 is now dead.
>>
>> but there is no other messages related to the nodes "dissappearing"  as far as I
can see in the 18 hours since that message occured.
>>
>> Restarting seems to recover things, but nodes seems to go away again (0.8 also seem
to be prone to commit logs being unreadable in some cases?)
>>
>> This is 0.8 build from trunk last Friday.
>>
>> I will try to enable some more debugging tomorrow to see if there is something interesting,
just curious if anyone else had noticed something like this.
>>
>> Regards,
>> Terje
>>
>>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message