Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 28FDF11F14 for ; Fri, 25 Apr 2014 15:38:37 +0000 (UTC) Received: (qmail 78947 invoked by uid 500); 25 Apr 2014 15:38:34 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 78917 invoked by uid 500); 25 Apr 2014 15:38:33 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 78909 invoked by uid 99); 25 Apr 2014 15:38:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Apr 2014 15:38:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of srobenal@stanford.edu designates 171.67.219.81 as permitted sender) Received: from [171.67.219.81] (HELO smtp.stanford.edu) (171.67.219.81) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Apr 2014 15:38:29 +0000 Received: from smtp.stanford.edu (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 0D37C22ABB for ; Fri, 25 Apr 2014 08:38:05 -0700 (PDT) Received: from mail-ob0-f171.google.com (mail-ob0-f171.google.com [209.85.214.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by smtp.stanford.edu (Postfix) with ESMTPS id DC08421931 for ; Fri, 25 Apr 2014 08:38:03 -0700 (PDT) Received: by mail-ob0-f171.google.com with SMTP id uy5so4442759obc.2 for ; Fri, 25 Apr 2014 08:38:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=ryrB/IBGnwgCo99QqBXgguir9KxslNMnpWvz4YsfQqI=; b=jUXbvQkoWxwtojSeSRs0Bd89xdGVj4PwZSnuGVorPGvQ43/4Qe/QwiSHx8vPerPCXU gGxeRlK+86Tx8b56BZnTDgWj2xl8VMSUoQcIz+7um/8vFp8yhhRlZa0cjcNlD2dPpeBZ nC4FgLiGhS+dsauML+IebRacsvA4SjnC1CWVVE26IDzfRNVQz43o6hlqpOMAs+ECtpyR 8FlNdmZoicz4FWjLaV8W7JPOdVO59IYPdJVYmUK0aJH3OclHYR6MAPO0/vsdNblEQ8MR vU133HYgbg5XtrDfmnP5urPlBnYzu6TDoyixg65ruohhv+QY9V2SA0LezQcZGAMtuHUr TDGg== X-Gm-Message-State: ALoCoQnnWFN2q5Ylb78KXfA7VqiF9LygbaN9FXKD5rXByfZsaaMLyao+NQKGoVW9iUM2jYJN6+ISBfUYykTBKXv5x3J4HK7glB1WKOGjn3/EcN4fwS0yi/vUVpQ06hLTi/kK6IzgVH+T X-Received: by 10.60.43.232 with SMTP id z8mr1949302oel.69.1398440283142; Fri, 25 Apr 2014 08:38:03 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.60.43.232 with SMTP id z8mr1949288oel.69.1398440283010; Fri, 25 Apr 2014 08:38:03 -0700 (PDT) Received: by 10.76.171.102 with HTTP; Fri, 25 Apr 2014 08:38:02 -0700 (PDT) In-Reply-To: References: Date: Fri, 25 Apr 2014 08:38:02 -0700 Message-ID: Subject: Re: Bootstrap Timing From: Steven A Robenalt To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113311fc4f0bb504f7dfc02a X-Virus-Checked: Checked by ClamAV on apache.org --001a113311fc4f0bb504f7dfc02a Content-Type: text/plain; charset=ISO-8859-1 Interesting. I did our 2.0.3 -> 2.0.5 upgrade by bootstrapping/joining each node into our cluster, one at a time, then retiring the old nodes one at a time. Maybe something specific to the 2.0.6 release? Good to hear that you've gotten through it anyway. Steve On Fri, Apr 25, 2014 at 7:49 AM, Phil Burress wrote: > Cassandra 2.0.6 > > > On Fri, Apr 25, 2014 at 10:31 AM, James Rothering wrote: > >> What version of C* is this? >> >> >> On Fri, Apr 25, 2014 at 6:55 AM, Phil Burress wrote: >> >>> Just a follow-up on this for any interested parties. Ultimately we've >>> determined that the bootstrap/join process is broken in Cassandra. We ended >>> up creating an entirely new cluster and migrating the data. >>> >>> >>> On Mon, Apr 21, 2014 at 10:32 AM, Phil Burress >> > wrote: >>> >>>> The new node has managed to stay up without dying for about 24 hours >>>> now... but it still is in JOINING state. A new concern has popped up. Disk >>>> usage is at 500GB on the new node. The three original nodes have about 40GB >>>> each. Any ideas why this is happening? >>>> >>>> >>>> On Sat, Apr 19, 2014 at 9:19 PM, Phil Burress >>> > wrote: >>>> >>>>> Thank you all for your advice and good info. The node has died a >>>>> couple of times with out of memory errors. I've restarted each time but it >>>>> starts re - running compaction and then dies again. >>>>> >>>>> Is there a better way to do this? >>>>> On Apr 18, 2014 6:06 PM, "Steven A Robenalt" >>>>> wrote: >>>>> >>>>>> That's what I'd be doing, but I wouldn't expect it to run for 3 days >>>>>> this time. My guess is that whatever was going wrong with the bootstrap >>>>>> when you had 3 nodes starting at once was interfering with the completion >>>>>> of the 1 remaining node of those 3. A clean bootstrap of a single node >>>>>> should complete eventually, and I would think it'll be a lot less than 3 >>>>>> days. Our database is much smaller than yours at the moment, so I can't >>>>>> really guide you on how long it should take, but I'd think that others on >>>>>> the list with similar database sizes might be able to give you a better >>>>>> idea. >>>>>> >>>>>> Steve >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Apr 18, 2014 at 1:43 PM, Phil Burress < >>>>>> philburresseme@gmail.com> wrote: >>>>>> >>>>>>> First, I just stopped 2 of the nodes and left one running. But this >>>>>>> morning, I stopped that third node, cleared out the data, restarted and let >>>>>>> it rejoin again. It appears streaming is done (according to netstats), >>>>>>> right now it appears to be running compaction and building secondary index >>>>>>> (according to compactionstats). Just sit and wait I guess? >>>>>>> >>>>>>> >>>>>>> On Fri, Apr 18, 2014 at 2:23 PM, Steven A Robenalt < >>>>>>> srobenal@stanford.edu> wrote: >>>>>>> >>>>>>>> Looking back through this email chain, it looks like Phil said he >>>>>>>> wasn't using vnodes. >>>>>>>> >>>>>>>> For the record, we are using vnodes since we brought up our first >>>>>>>> cluster, and have not seen any issues with bootstrapping new nodes either >>>>>>>> to replace existing nodes, or to grow/shrink the cluster. We did adhere to >>>>>>>> the caveats that new nodes should not be seed nodes, and that we should >>>>>>>> allow each node to join the cluster completely before making any other >>>>>>>> changes. >>>>>>>> >>>>>>>> Phil, when you dropped to adding just the single node to your >>>>>>>> cluster, did you start over with the newly added node (blowing away the >>>>>>>> database created on the previous startup), or did you shut down the other 2 >>>>>>>> added nodes and leave the remaining one in progress to continue? >>>>>>>> >>>>>>>> Steve >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Apr 18, 2014 at 10:40 AM, Robert Coli >>>>>>> > wrote: >>>>>>>> >>>>>>>>> On Fri, Apr 18, 2014 at 5:05 AM, Phil Burress < >>>>>>>>> philburresseme@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> nodetool netstats shows 84 files. They are all at 100%. Nothing >>>>>>>>>> showing in Pending or Active for Read Repair Stats. >>>>>>>>>> >>>>>>>>>> I'm assuming this means it's done. But it still shows "JOINING". >>>>>>>>>> Is there an undocumented step I'm missing here? This whole process seems >>>>>>>>>> broken to me. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Lately it seems like a lot more people than usual are : >>>>>>>>> >>>>>>>>> 1) using vnodes >>>>>>>>> 2) unable to bootstrap new nodes >>>>>>>>> >>>>>>>>> If I were you, I would likely file a JIRA detailing your negative >>>>>>>>> experience with this core functionality. >>>>>>>>> >>>>>>>>> =Rob >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Steve Robenalt >>>>>>>> Software Architect >>>>>>>> HighWire | Stanford University >>>>>>>> 425 Broadway St, Redwood City, CA 94063 >>>>>>>> >>>>>>>> srobenal@stanford.edu >>>>>>>> http://highwire.stanford.edu >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Steve Robenalt >>>>>> Software Architect >>>>>> HighWire | Stanford University >>>>>> 425 Broadway St, Redwood City, CA 94063 >>>>>> >>>>>> srobenal@stanford.edu >>>>>> http://highwire.stanford.edu >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>> >> > -- Steve Robenalt Software Architect HighWire | Stanford University 425 Broadway St, Redwood City, CA 94063 srobenal@stanford.edu http://highwire.stanford.edu --001a113311fc4f0bb504f7dfc02a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Interesting. I did our 2.0.3 -> 2.0.5 upgrade by bootst= rapping/joining each node into our cluster, one at a time, then retiring th= e old nodes one at a time. Maybe something specific to the 2.0.6 release?
Good to hear that you've gotten through it anyway.
=

Steve



On Fri, Apr 25, 2014 at 7:49 AM, Phi= l Burress <philburresseme@gmail.com> wrote:
Cassandra 2.0.6


On Fri, Apr 25, 2014 at 10:31 AM, James Rothering <jrothering@codojo.me= > wrote:
What version of C* is this?=


On Fri, Apr 25, 2014 at 6:55 AM, Phil Burress <philburresseme@gmail= .com> wrote:
Just a follow-up on this fo= r any interested parties. Ultimately we've determined that the bootstra= p/join process is broken in Cassandra. We ended up creating an entirely new= cluster and migrating the data.


On Mon, Apr 2= 1, 2014 at 10:32 AM, Phil Burress <philburresseme@gmail.com>= wrote:
The new node has managed to= stay up without dying for about 24 hours now... but it still is in JOINING= state. A new concern has popped up. Disk usage is at 500GB on the new node= . The three original nodes have about 40GB each. Any ideas why this is happ= ening?


On Sat, Apr 1= 9, 2014 at 9:19 PM, Phil Burress <philburresseme@gmail.com><= /span> wrote:

Thank you all for your advice= and good info. The node has died a couple of times with out of memory erro= rs. I've restarted each time but it starts re - running compaction and = then dies again.

Is there a better way to do this?

On Apr 18, 2014 6:06 PM, "Steven A Robenalt= " <srobe= nal@stanford.edu> wrote:
That's what I'd be doing, but I wouldn't expec= t it to run for 3 days this time. My guess is that whatever was going wrong= with the bootstrap when you had 3 nodes starting at once was interfering w= ith the completion of the 1 remaining node of those 3. A clean bootstrap of= a single node should complete eventually, and I would think it'll be a= lot less than 3 days. Our database is much smaller than yours at the momen= t, so I can't really guide you on how long it should take, but I'd = think that others on the list with similar database sizes might be able to = give you a better idea.

Steve

<= br>
On Fri, Apr 18, 2014 at 1:43 PM, Phil Bur= ress <philburresseme@gmail.com> wrote:
First, I just stopped 2 of = the nodes and left one running. But this morning, I stopped that third node= , cleared out the data, restarted and let it rejoin again. It appears strea= ming is done (according to netstats), right now it appears to be running co= mpaction and building secondary index (according to compactionstats). Just = sit and wait I guess?


On Fri, Apr 1= 8, 2014 at 2:23 PM, Steven A Robenalt <srobenal@stanford.edu> wrote:
Looking back through this e= mail chain, it looks like Phil said he wasn't using vnodes.

For the record, we are using vnodes since we brought up our first clus= ter, and have not seen any issues with bootstrapping new nodes either to re= place existing nodes, or to grow/shrink the cluster. We did adhere to the c= aveats that new nodes should not be seed nodes, and that we should allow ea= ch node to join the cluster completely before making any other changes.

Phil, when you dropped to adding just the single node t= o your cluster, did you start over with the newly added node (blowing away = the database created on the previous startup), or did you shut down the oth= er 2 added nodes and leave the remaining one in progress to continue?

Steve
<= br>
On Fri, Apr 18, 2014 at 10:40 AM, Robert = Coli <rcoli@eventbrite.com> wrote:
=
On Fri, Apr 18, 2014 at 5:05 AM, Phil Burre= ss <philburresseme@gmail.com> wrote:
nodetool netstats shows 84 = files. They are all at 100%. Nothing showing in Pending or Active for Read = Repair Stats.

I'm assuming this means it's done. But it still show= s "JOINING". Is there an undocumented step I'm missing here? = This whole process seems broken to me.

Lately it seems like a lot more people than usual are :

1) using vnodes
2) unable to bootstrap new= nodes

If I were you, I would likely file a JIRA d= etailing your negative experience with this core functionality.

=3DRob

=A0



<= font color=3D"#888888">--
Steve Robenalt
Software Architect
HighWire | Stanford University=A0=
425 Broadway St, Redwoo= d City, CA 94063=A0

srobenal@s= tanford.edu




--
=
Steve Robenalt
=
Software Architect
=
HighWire | Stanford University=A0
425 Broadway St, Redwoo= d City, CA 94063=A0

srobenal@s= tanford.edu







--
=
Steve Robenalt
=
Software Architect
=
HighWire | Stanford University=A0
425 Broadway St, Redwoo= d City, CA 94063=A0

srobenal@s= tanford.edu
--001a113311fc4f0bb504f7dfc02a--