Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of sdolgy@gmail.com designates
 209.85.210.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGYDNRsnLYfuTjbyGbL4m+3VgkcNmTt_rqOuwt_mc6PGCu235A@mail.gmail.com>
References: <71758072ACCC42D19AEBFE0380824EEA@gmail.com>
 <CAGYDNRsnLYfuTjbyGbL4m+3VgkcNmTt_rqOuwt_mc6PGCu235A@mail.gmail.com>
From: Sasha Dolgy <sdolgy@gmail.com>
Date: Wed, 13 Jul 2011 18:55:29 +0200
Message-ID: 
 <CAP5HrRb0Fa8C2oGBmWAnJ2t_HJhQJprs5hWmhzM90Zy1wD96OA@mail.gmail.com>
Subject: Re: One node down but it thinks its fine...
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

any firewall changes?  ping is fine ... but if you can't get from
node(a) to nodes(n) on the specific ports...

On Wed, Jul 13, 2011 at 6:47 PM, samal <samal@wakya.in> wrote:
> Check seed ip is same in all node and should not be loopback ip on cluste=
r.
>
> On Wed, Jul 13, 2011 at 8:40 PM, Ray Slakinski <ray.slakinski@gmail.com>
> wrote:
>>
>> One of our nodes, which happens to be the seed thinks its Up and all the
>> other nodes are down. However all the other nodes thinks the seed is dow=
n
>> instead. The logs for the seed node show everything is running as it sho=
uld
>> be. I've tried restarting the node, turning on/off gossip and thrift and
>> nothing seems to get the node to see the rest of its ring as up and runn=
ing.
>> I have also tried restarting one of the other nodes, which had no affect=
 on
>> the situation. Below is the ring outputs for the seed and one other node=
 in
>> the ring, plus a ping to show that the seed can ping the other node.
>>
>> # bin/nodetool -h 0.0.0.0 ring
>> Address Status State Load Owns Token
>> =A0141784319550391026443072753096570088105
>> 127.0.0.1 Up Normal 4.61 GB 16.67% 0
>> xx.xxx.30.210 Down Normal ? 16.67% 2835686391007820528861455061931401762=
1
>> xx.xx.90.87 Down Normal ? 16.67% 56713727820156410577229101238628035242
>> xx.xx.22.236 Down Normal ? 16.67% 85070591730234615865843651857942052863
>> xx.xx.97.96 Down Normal ? 16.67% 113427455640312821154458202477256070484
>> xx.xxx.17.122 Down Normal ? 16.67% 1417843195503910264430727530965700881=
05
>>
>>
>> # ping xx.xxx.30.210
>> PING xx.xxx.30.210 (xx.xxx.30.210) 56(84) bytes of data.
>> 64 bytes from xx.xxx.30.210: icmp_req=3D1 ttl=3D61 time=3D0.299 ms
>> 64 bytes from xx.xxx.30.210: icmp_req=3D2 ttl=3D61 time=3D0.287 ms
>> ^C
>> --- xx.xxx.30.210 ping statistics ---
>> 2 packets transmitted, 2 received, 0% packet loss, time 999ms
>> rtt min/avg/max/mdev =3D 0.287/0.293/0.299/0.006 ms
>>
>>
>> # bin/nodetool -h xx.xxx.30.210 ring
>> Address Status State Load Owns Token
>> =A0141784319550391026443072753096570088105
>> xx.xxx.23.40 Down Normal ? 16.67% 0
>> xx.xxx.30.210 Up Normal 10.58 GB 16.67%
>> 28356863910078205288614550619314017621
>> xx.xx.90.87 Up Normal 10.47 GB 16.67%
>> 56713727820156410577229101238628035242
>> xx.xx.22.236 Up Normal 9.63 GB 16.67%
>> 85070591730234615865843651857942052863
>> xx.xx.97.96 Up Normal 10.68 GB 16.67%
>> 113427455640312821154458202477256070484
>> xx.xxx.17.122 Up Normal 10.18 GB 16.67%
>> 141784319550391026443072753096570088105
>>
>> --
>> Ray Slakinski
>>
>>
>
>


--=20
Sasha Dolgy
sasha.dolgy@gmail.com