incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Reddy <mark.re...@boxever.com>
Subject Re: Node bootstrap
Date Tue, 05 Aug 2014 15:37:01 GMT
>
> Yes num_tokens is set to 256. initial_token is blank on all nodes
> including the new one.


Ok so you have num_tokens set to 256 for all nodes with initial_token
commented out, this means you are using vnodes and the new node will
automatically grab a list of tokens to take over responsibility for.

Pool Name                    Active   Pending      Completed   Blocked  All
> time blocked
> FlushWriter                       0         0           1136         0
>           512
>
> Looks like about 50% of flushes are blocked.
>

This is a problem as it indicates that the IO system cannot keep up.

Just ran this on the new node:
> nodetool netstats | grep "Streaming from" | wc -l
> 10


This is normal as the new node will most likely take tokens from all nodes
in the cluster.

Sorry for the multiple updates, but another thing I found was all the other
> existing nodes have themselves in the seeds list, but the new node does not
> have itself in the seeds list. Can that cause this issue?


Seeds are only used when a new node is bootstrapping into the cluster and
needs a set of ips to contact and discover the cluster, so this would have
no impact on data sizes or streaming. In general it would be considered
best practice to have a set of 2-3 seeds from each data center, with all
nodes having the same seed list.


What is the current output of 'nodetool compactionstats'? Could you also
paste the output of nodetool status <keyspace>?

Mark



On Tue, Aug 5, 2014 at 3:59 PM, Ruchir Jha <ruchir.jha@gmail.com> wrote:

> Sorry for the multiple updates, but another thing I found was all the
> other existing nodes have themselves in the seeds list, but the new node
> does not have itself in the seeds list. Can that cause this issue?
>
>
> On Tue, Aug 5, 2014 at 10:30 AM, Ruchir Jha <ruchir.jha@gmail.com> wrote:
>
>> Just ran this on the new node:
>>
>> nodetool netstats | grep "Streaming from" | wc -l
>> 10
>>
>> Seems like the new node is receiving data from 10 other nodes. Is that
>> expected in a vnodes enabled environment?
>>
>> Ruchir.
>>
>>
>>
>> On Tue, Aug 5, 2014 at 10:21 AM, Ruchir Jha <ruchir.jha@gmail.com> wrote:
>>
>>> Also not sure if this is relevant but just noticed the nodetool tpstats
>>> output:
>>>
>>> Pool Name                    Active   Pending      Completed   Blocked
>>>  All time blocked
>>> FlushWriter                       0         0           1136         0
>>>             512
>>>
>>> Looks like about 50% of flushes are blocked.
>>>
>>>
>>> On Tue, Aug 5, 2014 at 10:14 AM, Ruchir Jha <ruchir.jha@gmail.com>
>>> wrote:
>>>
>>>> Yes num_tokens is set to 256. initial_token is blank on all nodes
>>>> including the new one.
>>>>
>>>>
>>>> On Tue, Aug 5, 2014 at 10:03 AM, Mark Reddy <mark.reddy@boxever.com>
>>>> wrote:
>>>>
>>>>> My understanding was that if initial_token is left empty on the new
>>>>>> node, it just contacts the heaviest node and bisects its token range.
>>>>>
>>>>>
>>>>> If you are using vnodes and you have num_tokens set to 256 the new
>>>>> node will take token ranges dynamically. What is the configuration of
your
>>>>> other nodes, are you setting num_tokens or initial_token on those?
>>>>>
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>> On Tue, Aug 5, 2014 at 2:57 PM, Ruchir Jha <ruchir.jha@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Patricia for your response!
>>>>>>
>>>>>> On the new node, I just see a lot of the following:
>>>>>>
>>>>>> INFO [FlushWriter:75] 2014-08-05 09:53:04,394 Memtable.java (line
>>>>>> 400) Writing Memtable
>>>>>> INFO [CompactionExecutor:3] 2014-08-05 09:53:11,132
>>>>>> CompactionTask.java (line 262) Compacted 12 sstables to
>>>>>>
>>>>>> so basically it is just busy flushing, and compacting. Would you
have
>>>>>> any ideas on why the 2x disk space blow up. My understanding was
that if
>>>>>> initial_token is left empty on the new node, it just contacts the
heaviest
>>>>>> node and bisects its token range. And the heaviest node is around
2.1 TB,
>>>>>> and the new node is already at 4 TB. Could this be because compaction
is
>>>>>> falling behind?
>>>>>>
>>>>>> Ruchir
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 4, 2014 at 7:23 PM, Patricia Gorla <
>>>>>> patricia@thelastpickle.com> wrote:
>>>>>>
>>>>>>> Ruchir,
>>>>>>>
>>>>>>> What exactly are you seeing in the logs? Are you running major
>>>>>>> compactions on the new bootstrapping node?
>>>>>>>
>>>>>>> With respect to the seed list, it is generally advisable to use
3
>>>>>>> seed nodes per AZ / DC.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 4, 2014 at 11:41 AM, Ruchir Jha <ruchir.jha@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I am trying to bootstrap the thirteenth node in a 12 node
cluster
>>>>>>>> where the average data size per node is about 2.1 TB. The
bootstrap
>>>>>>>> streaming has been going on for 2 days now, and the disk
size on the new
>>>>>>>> node is already above 4 TB and still going. Is this because
the new node is
>>>>>>>> running major compactions while the streaming is going on?
>>>>>>>>
>>>>>>>> One thing that I noticed that seemed off was the seeds property
in
>>>>>>>> the yaml of the 13th node comprises of 1..12. Where as the
seeds property
>>>>>>>> on the existing 12 nodes consists of all the other nodes
except the
>>>>>>>> thirteenth node. Is this an issue?
>>>>>>>>
>>>>>>>> Any other insight is appreciated?
>>>>>>>>
>>>>>>>> Ruchir.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Patricia Gorla
>>>>>>> @patriciagorla
>>>>>>>
>>>>>>> Consultant
>>>>>>> Apache Cassandra Consulting
>>>>>>> http://www.thelastpickle.com <http://thelastpickle.com>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message