cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Di Spaltro <dan.dispal...@gmail.com>
Subject Re: Stalled Bootstrapping Process
Date Thu, 01 Apr 2010 19:22:20 GMT
But I didn't restart the red one.

On Thu, Apr 1, 2010 at 12:18 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> There shouldn't be anything to clean up.  (The temporary streaming
> files it anticompacted are automatically removed on restart)
>
> On Thu, Apr 1, 2010 at 2:17 PM, Dan Di Spaltro <dan.dispaltro@gmail.com>
> wrote:
> > Okay, so should I run any more commands like cleanup before?
> >
> > On Thu, Apr 1, 2010 at 12:09 PM, Jonathan Ellis <jbellis@gmail.com>
> wrote:
> >>
> >> Bootstrap source restarting will always fail bootstrap.  You'll need
> >> to restart the blue one too now, I'm afraid.
> >>
> >> On Thu, Apr 1, 2010 at 2:01 PM, Dan Di Spaltro <dan.dispaltro@gmail.com
> >
> >> wrote:
> >> > Before the Red one rebooted it had 1 active STREAM-STAGE.  Now it has
> 0
> >> > in
> >> > STREAM-STAGE.
> >> >
> >> > On Thu, Apr 1, 2010 at 11:57 AM, Dan Di Spaltro
> >> > <dan.dispaltro@gmail.com>
> >> > wrote:
> >> >>
> >> >> Red one.
> >> >> Gary - both say nothing is happening with no destinations or sources.
> >> >>
> >> >> On Thu, Apr 1, 2010 at 11:55 AM, Jonathan Ellis <jbellis@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> which node rebooted, the red one, or the blue one?
> >> >>>
> >> >>> On Thu, Apr 1, 2010 at 11:26 AM, Dan Di Spaltro
> >> >>> <dan.dispaltro@gmail.com>
> >> >>> wrote:
> >> >>> > So we are adding another node to the cluster with the latest
0.6
> >> >>> > branch
> >> >>> > (RC1).  It seems to be hung in some limbo state.
> >> >>> > Before bootstrapping our cluster had 50-60GB spread fairly
evenly
> >> >>> > across 4
> >> >>> > machines, with RF=3.   One machine had more load than the
others,
> >> >>> > and
> >> >>> > sure
> >> >>> > enough bootstrapping selected that node.   That is the red
> machine.
> >> >>> >  The
> >> >>> > light blue machine is the new machine.
> >> >>> > I have attached a graph to illustrate when the bootstrap process
> >> >>> > started.
> >> >>> > In jconsole the streamingservice status was "performing
> >> >>> > anticompaction..."
> >> >>> > for over 18-24 hrs.  It is currently in "nothing is happening".
> It
> >> >>> > did
> >> >>> > have 1 active STREAM-STAGE task, but the machine had to be
> rebooted
> >> >>> > for
> >> >>> > something unrelated to cassandra. Now the light blue machine
> appears
> >> >>> > to
> >> >>> > be
> >> >>> > getting data, but its growing at virtually the same rate as
the
> >> >>> > other
> >> >>> > machines which makes me think it is part of the cluster and
not
> >> >>> > actually
> >> >>> > streaming data from the machine its supposed to.
> >> >>> > Any other ideas on how to debug?
> >> >>> >
> >> >>> > --
> >> >>> > Dan Di Spaltro
> >> >>> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Dan Di Spaltro
> >> >
> >> >
> >> >
> >> > --
> >> > Dan Di Spaltro
> >> >
> >
> >
> >
> > --
> > Dan Di Spaltro
> >
>



-- 
Dan Di Spaltro

Mime
View raw message