From user-return-7664-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Fri Jul 16 05:45:51 2010 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 93433 invoked from network); 16 Jul 2010 05:45:51 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 16 Jul 2010 05:45:51 -0000 Received: (qmail 29088 invoked by uid 500); 16 Jul 2010 05:45:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 28832 invoked by uid 500); 16 Jul 2010 05:45:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 28824 invoked by uid 99); 16 Jul 2010 05:45:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 05:45:46 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [131.215.239.119] (HELO mail.alumni.caltech.edu) (131.215.239.119) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 05:45:39 +0000 Received: from localhost (dsl081-082-089.lax1.dsl.speakeasy.net [64.81.82.89]) by mail.alumni.caltech.edu (Postfix) with ESMTPSA id 41DB33F0839; Thu, 15 Jul 2010 22:45:09 -0700 (PDT) X-DKIM: Sendmail DKIM Filter v2.8.2 mail.alumni.caltech.edu 41DB33F0839 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=alumni.caltech.edu; s=enforce; t=1279259109; bh=pu4pv1ejGwOkhjSDYJZJ06MDc7YEPc9qAOy2mLp1neM=; h=Date:From:To:Subject:Message-ID:References:Mime-Version: Content-Type:Content-Transfer-Encoding:In-Reply-To; b=JwCOabIc59VsRtwPIlvBltNokYWgMobUNq1JSGT0A943R6Y/bGzgAzAs57hKJ6RtH 9Q/FLJNafq6Z/ZkB9TnKEZa+XEJhuvwkZEe6pgAPu7e5xh62yVZ9Mj5VTnMYMBK+LB RerjStALbDbPALqy8v9ylqt8vrNJIvm29mXJ7KNg= Date: Thu, 15 Jul 2010 22:45:08 -0700 From: Anthony Molinaro To: user@cassandra.apache.org Subject: Re: Bootstrap question Message-ID: <20100716054508.GA73522@alumni.caltech.edu> Mail-Followup-To: user@cassandra.apache.org References: <20100714225847.GA64220@alumni.caltech.edu> <20100715202806.GB71234@alumni.caltech.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.4.2.3i X-MailScanner-Information-Alumni: X-Alumni-MailScanner-ID: 41DB33F0839.ABB47 X-MailScanner-Alumni: No Virii found X-Spam-Status-Alumni: not spam, SpamAssassin (not cached, score=-4.398, required 5, autolearn=not spam, ALL_TRUSTED -1.80, BAYES_00 -2.60, DKIM_SIGNED 0.00) X-MailScanner-From: anthonym@alumni.caltech.edu X-Virus-Checked: Checked by ClamAV on apache.org Okay, so things were pretty messed up. I shut down all the new nodes, then the old nodes started doing the half the ring is down garbage which pretty much requires a full restart of everything. So I had to shut everything down, then bring the seed back, then the rest of the nodes, so they finally all agreed on the ring again. Then I started one of the new nodes, and have been watching the logs, so far 2 hours since the "Bootstrapping" message appeared in the new log and nothing has happened. No anticompaction messages anywhere, there's one node compacting, but its on the other end of the ring, so no where near that new node. I'm wondering if it will ever get data at this point. Is there something else I should try? The only thing I can think of is deleting the system directory on the new node, and restarting, so I'll try that and see if it does anything. -Anthony On Thu, Jul 15, 2010 at 03:43:49PM -0500, Jonathan Ellis wrote: > On Thu, Jul 15, 2010 at 3:28 PM, Anthony Molinaro > wrote: > > Is the fact that 2 new nodes are in the range messing it up? > > Probably. > > >  And if so > > how do I recover (I'm thinking, shutdown new nodes 2,3,4,5, the bringing > > up nodes 2,4, waiting for them to finish, then bringing up 3,5?). > > Yes. > > You might have to restart the old nodes too to clear out the confusion. > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com -- ------------------------------------------------------------------------ Anthony Molinaro