cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Stalled Bootstrapping Process
Date Thu, 01 Apr 2010 19:18:48 GMT
There shouldn't be anything to clean up.  (The temporary streaming
files it anticompacted are automatically removed on restart)

On Thu, Apr 1, 2010 at 2:17 PM, Dan Di Spaltro <dan.dispaltro@gmail.com> wrote:
> Okay, so should I run any more commands like cleanup before?
>
> On Thu, Apr 1, 2010 at 12:09 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> Bootstrap source restarting will always fail bootstrap.  You'll need
>> to restart the blue one too now, I'm afraid.
>>
>> On Thu, Apr 1, 2010 at 2:01 PM, Dan Di Spaltro <dan.dispaltro@gmail.com>
>> wrote:
>> > Before the Red one rebooted it had 1 active STREAM-STAGE.  Now it has 0
>> > in
>> > STREAM-STAGE.
>> >
>> > On Thu, Apr 1, 2010 at 11:57 AM, Dan Di Spaltro
>> > <dan.dispaltro@gmail.com>
>> > wrote:
>> >>
>> >> Red one.
>> >> Gary - both say nothing is happening with no destinations or sources.
>> >>
>> >> On Thu, Apr 1, 2010 at 11:55 AM, Jonathan Ellis <jbellis@gmail.com>
>> >> wrote:
>> >>>
>> >>> which node rebooted, the red one, or the blue one?
>> >>>
>> >>> On Thu, Apr 1, 2010 at 11:26 AM, Dan Di Spaltro
>> >>> <dan.dispaltro@gmail.com>
>> >>> wrote:
>> >>> > So we are adding another node to the cluster with the latest 0.6
>> >>> > branch
>> >>> > (RC1).  It seems to be hung in some limbo state.
>> >>> > Before bootstrapping our cluster had 50-60GB spread fairly evenly
>> >>> > across 4
>> >>> > machines, with RF=3.   One machine had more load than the others,
>> >>> > and
>> >>> > sure
>> >>> > enough bootstrapping selected that node.   That is the red machine.
>> >>> >  The
>> >>> > light blue machine is the new machine.
>> >>> > I have attached a graph to illustrate when the bootstrap process
>> >>> > started.
>> >>> > In jconsole the streamingservice status was "performing
>> >>> > anticompaction..."
>> >>> > for over 18-24 hrs.  It is currently in "nothing is happening".
  It
>> >>> > did
>> >>> > have 1 active STREAM-STAGE task, but the machine had to be rebooted
>> >>> > for
>> >>> > something unrelated to cassandra. Now the light blue machine appears
>> >>> > to
>> >>> > be
>> >>> > getting data, but its growing at virtually the same rate as the
>> >>> > other
>> >>> > machines which makes me think it is part of the cluster and not
>> >>> > actually
>> >>> > streaming data from the machine its supposed to.
>> >>> > Any other ideas on how to debug?
>> >>> >
>> >>> > --
>> >>> > Dan Di Spaltro
>> >>> >
>> >>
>> >>
>> >>
>> >> --
>> >> Dan Di Spaltro
>> >
>> >
>> >
>> > --
>> > Dan Di Spaltro
>> >
>
>
>
> --
> Dan Di Spaltro
>

Mime
View raw message