cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Molinaro <antho...@alumni.caltech.edu>
Subject Re: Odd ring problems with 0.5.1
Date Fri, 23 Apr 2010 21:22:11 GMT

On Fri, Apr 23, 2010 at 01:17:21PM -0500, Jonathan Ellis wrote:
> On Fri, Apr 23, 2010 at 1:12 PM, Anthony Molinaro
> <anthonym@alumni.caltech.edu> wrote:
> > I'm not sure how it would get this, maybe I need to restart my seed node?
> 
> It's worth a try.  Sounds like you found an unusual bug in gossip.

Damn, restarting the seed, resulted in the seed coming up in a new ring
with 3 nodes which have been decommissioned.  Seems like restarting other
nodes brings them into that ring (or at least the first few seem to be in
the new ring).  I'll restart them all to see if I can't get to a consistent
ring.  You know what might have happened, I changed the ip of the seed host
in my /etc/hosts before starting to decommission, I bet I should have then
restarted everything.  Oh well, hopefully most of my data is still viable.

I do still have all the old sstables lying around, can I just sstable2json
then json2sstable and have it reload them?  Or do the sstables need to be
keyed to the keyrange?  I guess I can sstable2json then create an import
script to insert them via thrift?

> > When I run nodeprobe ring on the seed I don't see any of the hosts I
> > decommissioned, but maybe they are still listed there somewhere?
> 
> 0.5 does leave decommissioned host information in gossip, but I'm not
> sure how that applies to this problem.

I bet that was a red herring, I'm pretty convinced now this was all a
result of me now restarting all the nodes after making a change to the
seed.

-Anthony

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@alumni.caltech.edu>

Mime
View raw message