incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Molinaro <antho...@alumni.caltech.edu>
Subject Re: Odd ring problems with 0.5.1
Date Sat, 24 Apr 2010 01:55:05 GMT
Turns out I needed to shut everything down completely, then start it all up
a rolling restart was still resulting in some nodes being confused about
what ring they were in.

I think the moral of all this, is any changes to the seed node must result
in a full restart of your cluster.  Also any use of removetoken is perilous.

Good news is I'm off of the old nodes, I'll need to figure out a way to
bulk load the data from some of the old sstables, but I think sstable2json
and a quick perl script to load might work out.

Then after that upgrade to 0.6.x

-Anthony

On Fri, Apr 23, 2010 at 02:22:11PM -0700, Anthony Molinaro wrote:
> 
> On Fri, Apr 23, 2010 at 01:17:21PM -0500, Jonathan Ellis wrote:
> > On Fri, Apr 23, 2010 at 1:12 PM, Anthony Molinaro
> > <anthonym@alumni.caltech.edu> wrote:
> > > I'm not sure how it would get this, maybe I need to restart my seed node?
> > 
> > It's worth a try.  Sounds like you found an unusual bug in gossip.
> 
> Damn, restarting the seed, resulted in the seed coming up in a new ring
> with 3 nodes which have been decommissioned.  Seems like restarting other
> nodes brings them into that ring (or at least the first few seem to be in
> the new ring).  I'll restart them all to see if I can't get to a consistent
> ring.  You know what might have happened, I changed the ip of the seed host
> in my /etc/hosts before starting to decommission, I bet I should have then
> restarted everything.  Oh well, hopefully most of my data is still viable.
> 
> I do still have all the old sstables lying around, can I just sstable2json
> then json2sstable and have it reload them?  Or do the sstables need to be
> keyed to the keyrange?  I guess I can sstable2json then create an import
> script to insert them via thrift?
> 
> > > When I run nodeprobe ring on the seed I don't see any of the hosts I
> > > decommissioned, but maybe they are still listed there somewhere?
> > 
> > 0.5 does leave decommissioned host information in gossip, but I'm not
> > sure how that applies to this problem.
> 
> I bet that was a red herring, I'm pretty convinced now this was all a
> result of me now restarting all the nodes after making a change to the
> seed.
> 
> -Anthony
> 
> -- 
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym@alumni.caltech.edu>

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@alumni.caltech.edu>

Mime
View raw message