cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Tavory <ran...@gmail.com>
Subject Re: Bootstrapping taking long
Date Wed, 05 Jan 2011 16:01:16 GMT
OK, thanks, so I see we had the same problem (I too had multiple keyspace,
not that I know why it matters to the problem at hand) and I see that by
upgrading to 0.6.7 you solved your problem (I didn't try it, had a different
workaround) but frankly, I don't understand how
https://issues.apache.org/jira/browse/CASSANDRA-1676 would relate the the
"stuck bootstrap" problem (I'm not saying that it isn't, I'd just like to
understand why...)


On Wed, Jan 5, 2011 at 5:42 PM, Thibaut Britz <thibaut.britz@trendiction.com
> wrote:

> Had the same Problem a while ago. Upgrading solved the problem (Don't know
> if you have to redeploy your cluster though)
>
> http://www.mail-archive.com/user@cassandra.apache.org/msg07106.html
>
>
>
> On Wed, Jan 5, 2011 at 4:29 PM, Ran Tavory <rantav@gmail.com> wrote:
>
>> @Thibaut wrong email? Or how's "Avoid dropping messages off the client
>> request path" (CASSANDRA-1676) related to the bootstrap questions I had?
>>
>>
>> On Wed, Jan 5, 2011 at 5:23 PM, Thibaut Britz <
>> thibaut.britz@trendiction.com> wrote:
>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-1676
>>>
>>> you have to use at least 0.6.7
>>>
>>>
>>>
>>> On Wed, Jan 5, 2011 at 4:19 PM, Edward Capriolo <edlinuxguru@gmail.com>wrote:
>>>
>>>> On Wed, Jan 5, 2011 at 10:05 AM, Ran Tavory <rantav@gmail.com> wrote:
>>>> > In storage-conf I see this comment [1] from which I understand that
>>>> the
>>>> > recommended way to bootstrap a new node is to set AutoBootstrap=true
>>>> and
>>>> > remove itself from the seeds list.
>>>> > Moreover, I did try to set AutoBootstrap=true and have the node in its
>>>> own
>>>> > seeds list, but it would not bootstrap. I don't recall the exact
>>>> message but
>>>> > it was something like "I found myself in the seeds list therefore I'm
>>>> not
>>>> > going to bootstrap even though AutoBootstrap is true".
>>>> >
>>>> > [1]
>>>> >   <!--
>>>> >    ~ Turn on to make new [non-seed] nodes automatically migrate the
>>>> right
>>>> > data
>>>> >    ~ to themselves.  (If no InitialToken is specified, they will pick
>>>> one
>>>> >    ~ such that they will get half the range of the most-loaded node.)
>>>> >    ~ If a node starts up without bootstrapping, it will mark itself
>>>> > bootstrapped
>>>> >    ~ so that you can't subsequently accidently bootstrap a node with
>>>> >    ~ data on it.  (You can reset this by wiping your data and
>>>> commitlog
>>>> >    ~ directories.)
>>>> >    ~
>>>> >    ~ Off by default so that new clusters and upgraders from 0.4 don't
>>>> >    ~ bootstrap immediately.  You should turn this on when you start
>>>> adding
>>>> >    ~ new nodes to a cluster that already has data on it.  (If you are
>>>> > upgrading
>>>> >    ~ from 0.4, start your cluster with it off once before changing it
>>>> to
>>>> > true.
>>>> >    ~ Otherwise, no data will be lost but you will incur a lot of
>>>> unnecessary
>>>> >    ~ I/O before your cluster starts up.)
>>>> >   -->
>>>> >   <AutoBootstrap>false</AutoBootstrap>
>>>> > On Wed, Jan 5, 2011 at 4:58 PM, David Boxenhorn <david@lookin2.com>
>>>> wrote:
>>>> >>
>>>> >> If "seed list should be the same across the cluster" that means
that
>>>> nodes
>>>> >> *should* have themselves as a seed. If that doesn't work for Ran,
>>>> then that
>>>> >> is the first problem, no?
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 5, 2011 at 3:56 PM, Jake Luciani <jakers@gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>> Well your ring issues don't make sense to me, seed list should
be
>>>> the
>>>> >>> same across the cluster.
>>>> >>> I'm just thinking of other things to try, non-boostrapped nodes
>>>> should
>>>> >>> join the ring instantly but reads will fail if you aren't using
>>>> quorum.
>>>> >>>
>>>> >>> On Wed, Jan 5, 2011 at 8:51 AM, Ran Tavory <rantav@gmail.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> I haven't tried repair.  Should I?
>>>> >>>>
>>>> >>>> On Jan 5, 2011 3:48 PM, "Jake Luciani" <jakers@gmail.com>
wrote:
>>>> >>>> > Have you tried not bootstrapping but setting the token
and
>>>> manually
>>>> >>>> > calling
>>>> >>>> > repair?
>>>> >>>> >
>>>> >>>> > On Wed, Jan 5, 2011 at 7:07 AM, Ran Tavory <rantav@gmail.com>
>>>> wrote:
>>>> >>>> >
>>>> >>>> >> My conclusion is lame: I tried this on several
hosts and saw the
>>>> same
>>>> >>>> >> behavior, the only way I was able to join new nodes
was to first
>>>> >>>> >> start them
>>>> >>>> >> when they are *not in* their own seeds list and
after they
>>>> >>>> >> finish transferring the data, then restart them
with themselves
>>>> *in*
>>>> >>>> >> their
>>>> >>>> >> own seeds list. After doing that the node would
join the ring.
>>>> >>>> >> This is either my misunderstanding or a bug, but
the only place
>>>> I
>>>> >>>> >> found it
>>>> >>>> >> documented stated that the new node should not
be in its own
>>>> seeds
>>>> >>>> >> list.
>>>> >>>> >> Version 0.6.6.
>>>> >>>> >>
>>>> >>>> >> On Wed, Jan 5, 2011 at 10:35 AM, David Boxenhorn
>>>> >>>> >> <david@lookin2.com>wrote:
>>>> >>>> >>
>>>> >>>> >>> My nodes all have themselves in their list
of seeds - always
>>>> did -
>>>> >>>> >>> and
>>>> >>>> >>> everything works. (You may ask why I did this.
I don't know, I
>>>> must
>>>> >>>> >>> have
>>>> >>>> >>> copied it from an example somewhere.)
>>>> >>>> >>>
>>>> >>>> >>> On Wed, Jan 5, 2011 at 9:42 AM, Ran Tavory
<rantav@gmail.com>
>>>> wrote:
>>>> >>>> >>>
>>>> >>>> >>>> I was able to make the node join the ring
but I'm confused.
>>>> >>>> >>>> What I did is, first when adding the node,
this node was not
>>>> in the
>>>> >>>> >>>> seeds
>>>> >>>> >>>> list of itself. AFAIK this is how it's
supposed to be. So it
>>>> was
>>>> >>>> >>>> able to
>>>> >>>> >>>> transfer all data to itself from other
nodes but then it
>>>> stayed in
>>>> >>>> >>>> the
>>>> >>>> >>>> bootstrapping state.
>>>> >>>> >>>> So what I did (and I don't know why it
works), is add this
>>>> node to
>>>> >>>> >>>> the
>>>> >>>> >>>> seeds list in its own storage-conf.xml
file. Then restart the
>>>> >>>> >>>> server and
>>>> >>>> >>>> then I finally see it in the ring...
>>>> >>>> >>>> If I had added the node to the seeds list
of itself when first
>>>> >>>> >>>> joining
>>>> >>>> >>>> it, it would not join the ring but if I
do it in two phases it
>>>> did
>>>> >>>> >>>> work.
>>>> >>>> >>>> So it's either my misunderstanding or a
bug...
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>> On Wed, Jan 5, 2011 at 7:14 AM, Ran Tavory
<rantav@gmail.com>
>>>> >>>> >>>> wrote:
>>>> >>>> >>>>
>>>> >>>> >>>>> The new node does not see itself as
part of the ring, it sees
>>>> all
>>>> >>>> >>>>> others
>>>> >>>> >>>>> but itself, so from that perspective
the view is consistent.
>>>> >>>> >>>>> The only problem is that the node never
finishes to
>>>> bootstrap. It
>>>> >>>> >>>>> stays
>>>> >>>> >>>>> in this state for hours (It's been
20 hours now...)
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>> $ bin/nodetool -p 9004 -h localhost
streams
>>>> >>>> >>>>>> Mode: Bootstrapping
>>>> >>>> >>>>>> Not sending any streams.
>>>> >>>> >>>>>> Not receiving any streams.
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>> On Wed, Jan 5, 2011 at 1:20 AM, Nate
McCall <
>>>> nate@riptano.com>
>>>> >>>> >>>>> wrote:
>>>> >>>> >>>>>
>>>> >>>> >>>>>> Does the new node have itself in
the list of seeds per
>>>> chance?
>>>> >>>> >>>>>> This
>>>> >>>> >>>>>> could cause some issues if so.
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> On Tue, Jan 4, 2011 at 4:10 PM,
Ran Tavory <
>>>> rantav@gmail.com>
>>>> >>>> >>>>>> wrote:
>>>> >>>> >>>>>> > I'm still at lost. I haven't
been able to resolve this. I
>>>> tried
>>>> >>>> >>>>>> > adding another node at a different
location on the ring
>>>> but
>>>> >>>> >>>>>> > this node
>>>> >>>> >>>>>> > too remains stuck in the bootstrapping
state for many
>>>> hours
>>>> >>>> >>>>>> > without
>>>> >>>> >>>>>> > any of the other nodes being
busy with anti compaction or
>>>> >>>> >>>>>> > anything
>>>> >>>> >>>>>> > else. I don't know what's
keeping it from finishing the
>>>> >>>> >>>>>> > bootstrap,no
>>>> >>>> >>>>>> > CPU, no io, files were already
streamed so what is it
>>>> waiting
>>>> >>>> >>>>>> > for?
>>>> >>>> >>>>>> > I read the release notes of
0.6.7 and 0.6.8 and there
>>>> didn't
>>>> >>>> >>>>>> > seem to
>>>> >>>> >>>>>> > be anything addressing a similar
issue so I figured there
>>>> was
>>>> >>>> >>>>>> > no
>>>> >>>> >>>>>> point
>>>> >>>> >>>>>> > in upgrading. But let me know
if you think there is.
>>>> >>>> >>>>>> > Or any other advice...
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> > On Tuesday, January 4, 2011,
Ran Tavory <rantav@gmail.com
>>>> >
>>>> >>>> >>>>>> > wrote:
>>>> >>>> >>>>>> >> Thanks Jake, but unfortunately
the streams directory is
>>>> empty
>>>> >>>> >>>>>> >> so I
>>>> >>>> >>>>>> don't think that any of the nodes
is anti-compacting data
>>>> right
>>>> >>>> >>>>>> now or had
>>>> >>>> >>>>>> been in the past 5 hours. It seems
that all the data was
>>>> already
>>>> >>>> >>>>>> transferred
>>>> >>>> >>>>>> to the joining host but the joining
node, after having
>>>> received
>>>> >>>> >>>>>> the data
>>>> >>>> >>>>>> would still remain in bootstrapping
mode and not join the
>>>> >>>> >>>>>> cluster. I'm not
>>>> >>>> >>>>>> sure that *all* data was transferred
(perhaps other nodes
>>>> need to
>>>> >>>> >>>>>> transfer
>>>> >>>> >>>>>> more data) but nothing is actually
happening so I assume all
>>>> has
>>>> >>>> >>>>>> been moved.
>>>> >>>> >>>>>> >> Perhaps it's a configuration
error from my part. Should I
>>>> use
>>>> >>>> >>>>>> >> I use
>>>> >>>> >>>>>> AutoBootstrap=true ? Anything else
I should look out for in
>>>> the
>>>> >>>> >>>>>> configuration file or something
else?
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> On Tue, Jan 4, 2011 at
4:08 PM, Jake Luciani
>>>> >>>> >>>>>> >> <jakers@gmail.com>
>>>> >>>> >>>>>> wrote:
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> In 0.6, locate the node
doing anti-compaction and look in
>>>> the
>>>> >>>> >>>>>> "streams" subdirectory in the keyspace
data dir to monitor
>>>> the
>>>> >>>> >>>>>> anti-compaction progress (it puts
new SSTables for
>>>> bootstrapping
>>>> >>>> >>>>>> node in
>>>> >>>> >>>>>> there)
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> On Tue, Jan 4, 2011 at
8:01 AM, Ran Tavory <
>>>> rantav@gmail.com>
>>>> >>>> >>>>>> wrote:
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> Running nodetool decommission
didn't help. Actually the
>>>> node
>>>> >>>> >>>>>> >> refused
>>>> >>>> >>>>>> to decommission itself (b/c it
wasn't part of the ring). So
>>>> I
>>>> >>>> >>>>>> simply stopped
>>>> >>>> >>>>>> the process, deleted all the data
directories and started it
>>>> >>>> >>>>>> again. It
>>>> >>>> >>>>>> worked in the sense of the node
bootstrapped again but as
>>>> before,
>>>> >>>> >>>>>> after it
>>>> >>>> >>>>>> had finished moving the data nothing
happened for a long
>>>> time
>>>> >>>> >>>>>> (I'm still
>>>> >>>> >>>>>> waiting, but nothing seems to be
happening).
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> Any hints how to analyze
a "stuck" bootstrapping
>>>> node??thanks
>>>> >>>> >>>>>> >> On Tue, Jan 4, 2011 at
1:51 PM, Ran Tavory <
>>>> rantav@gmail.com>
>>>> >>>> >>>>>> wrote:
>>>> >>>> >>>>>> >> Thanks Shimi, so indeed
anticompaction was run on one of
>>>> the
>>>> >>>> >>>>>> >> other
>>>> >>>> >>>>>> nodes from the same DC but to my
understanding it has
>>>> already
>>>> >>>> >>>>>> ended. A few
>>>> >>>> >>>>>> hour ago...
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> I plenty of log messages
such as [1] which ended a couple
>>>> of
>>>> >>>> >>>>>> >> hours
>>>> >>>> >>>>>> ago, and I've seen the new node
streaming and accepting the
>>>> data
>>>> >>>> >>>>>> from the
>>>> >>>> >>>>>> node which performed the anticompaction
and so far it was
>>>> normal
>>>> >>>> >>>>>> so it
>>>> >>>> >>>>>> seemed that data is at its right
place. But now the new node
>>>> >>>> >>>>>> seems sort of
>>>> >>>> >>>>>> stuck. None of the other nodes
is anticompacting right now
>>>> or had
>>>> >>>> >>>>>> been
>>>> >>>> >>>>>> anticompacting since then.
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> The new node's CPU is
close to zero, it's iostats are
>>>> almost
>>>> >>>> >>>>>> >> zero so
>>>> >>>> >>>>>> I can't find another bottleneck
that would keep it hanging.
>>>> >>>> >>>>>> >> On the IRC someone suggested
I'd maybe retry to join this
>>>> >>>> >>>>>> >> node,
>>>> >>>> >>>>>> e.g. decommission and rejoin it
again. I'll try it now...
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> [1] INFO [COMPACTION-POOL:1]
2011-01-04 04:04:09,721
>>>> >>>> >>>>>> CompactionManager.java (line 338)
AntiCompacting
>>>> >>>> >>>>>>
>>>> >>>> >>>>>>
>>>> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvAds-6449-Data.db')]
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> INFO [COMPACTION-POOL:1]
2011-01-04 04:34:18,683
>>>> >>>> >>>>>> CompactionManager.java (line 338)
AntiCompacting
>>>> >>>> >>>>>>
>>>> >>>> >>>>>>
>>>> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3874-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3873-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3876-Data.db')]
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> INFO [COMPACTION-POOL:1]
2011-01-04 04:34:19,132
>>>> >>>> >>>>>> CompactionManager.java (line 338)
AntiCompacting
>>>> >>>> >>>>>>
>>>> >>>> >>>>>>
>>>> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvRatings-951-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvRatings-976-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvRatings-978-Data.db')]
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> INFO [COMPACTION-POOL:1]
2011-01-04 04:34:26,486
>>>> >>>> >>>>>> CompactionManager.java (line 338)
AntiCompacting
>>>> >>>> >>>>>>
>>>> >>>> >>>>>>
>>>> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvAds-6449-Data.db')]
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> On Tue, Jan 4, 2011 at
12:45 PM, shimi <
>>>> shimi.k@gmail.com>
>>>> >>>> >>>>>> >> wrote:
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> In my experience most
of the time it takes for a node to
>>>> join
>>>> >>>> >>>>>> >> the
>>>> >>>> >>>>>> cluster is the anticompaction on
the other nodes. The
>>>> streaming
>>>> >>>> >>>>>> part is very
>>>> >>>> >>>>>> fast.
>>>> >>>> >>>>>> >> Check the other nodes
logs to see if there is any node
>>>> doing
>>>> >>>> >>>>>> anticompaction.I don't remember
how much data I had in the
>>>> >>>> >>>>>> cluster when I
>>>> >>>> >>>>>> needed to add/remove nodes. I do
remember that it took a few
>>>> >>>> >>>>>> hours.
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> The node will join the
ring only when it will finish the
>>>> >>>> >>>>>> >> bootstrap.
>>>> >>>> >>>>>> >> --
>>>> >>>> >>>>>> >> /Ran
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> > --
>>>> >>>> >>>>>> > /Ran
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>> --
>>>> >>>> >>>>> /Ran
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>> --
>>>> >>>> >>>> /Ran
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >> --
>>>> >>>> >> /Ran
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > /Ran
>>>> >
>>>>
>>>> If non-auto-bootstrap nodes to not join they check to make sure good
>>>> old iptables is not on.
>>>>
>>>> Edward
>>>>
>>>
>>>
>>
>>
>> --
>> /Ran
>>
>>
>


-- 
/Ran

Mime
View raw message