Is there a way to force gossip among the nodes?
Subject: RE: gossip not working
Date: Thu, 4 Apr 2013 19:59:45 -0500I am not seeing anything in the logs other than "Starting up server gossip"and there is no firewall between the nodes.
Subject: Re: gossip not working
Date: Thu, 4 Apr 2013 18:49:29 -0500
What errors are you seeing in the log files of the down nodes? Did you run upgradesstables? You need to upgradesstables when moving from < 1.1.7 to 1.1.9On Apr 4, 2013, at 6:11 PM, S C <email@example.com> wrote:I was in the middle of upgrade to 1.1.9. I brought one node with 1.1.9 while the other were running on 1.1.5. Once one of the node was on 1.1.9 it is no longer recognizing other nodes in the ring.On 192.168.56.10 and 11192.168.56.10 DC1-Cass RAC1 Up Normal 28.06 GB 50.00% 0192.168.56.11 DC1-Cass RAC1 Up Normal 31.59 GB 25.00% 42535295865117307932921825928971026432192.168.56.12 DC1-Cass RAC1 Down Normal 29.02 GB 25.00% 85070591730234615865843651857942052864On 192.168.56.12122.214.171.124 DC1-Cass RAC1 Down Normal 28.06 GB 50.00% 0192.168.56.11 DC1-Cass RAC1 Down Normal 31.59 GB 25.00% 42535295865117307932921825928971026432192.168.56.12 DC1-Cass RAC1 Up Normal 29.02 GB 25.00% 85070591730234615865843651857942052864I do not see anything in the logs that tells me that there is a gossip issue.nodetool infoToken : 85070591730234615865843651857942052864Gossip active : trueThrift active : trueLoad : 29.05 GBGeneration No : 1365114563Uptime (seconds) : 2127Heap Memory (MB) : 848.71 / 7945.94Exceptions : 0Key Cache : size 2208 (bytes), capacity 104857584 (bytes), 1056 hits, 1099 requests, 0.961 recent hit rate, 14400 save period in secondsRow Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in secondsnodetool infoToken : 42535295865117307932921825928971026432Gossip active : trueThrift active : trueLoad : 31.59 GBGeneration No : 1364413038Uptime (seconds) : 703904Heap Memory (MB) : 733.02 / 7945.94Exceptions : 1Key Cache : size 3693312 (bytes), capacity 104857584 (bytes), 26071678 hits, 26616282 requests, 0.980 recent hit rate, 14400 save period in secondsRow Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in secondsThere is no firewall between the nodes and I can reach each other on storage port.What else should I be looking at to find root cause? Appreciate your inputs.