incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Watson <j...@disqus.com>
Subject Re: Adding nodes in 1.2 with vnodes requires huge disks
Date Mon, 29 Apr 2013 17:41:22 GMT
They were all restarted a couple times after adding 'num_tokens: 256' to
cassandra.yaml.

Yes and nodetool ring became 'unusable' due to all the new tokens.


On Mon, Apr 29, 2013 at 10:24 AM, Sam Overton <sam@acunu.com> wrote:

> Did you update num_tokens on the existing hosts and restart them, before
> you tried bootstrapping in the new node? If the new node tried to stream
> all the data in the cluster then this would be consistent with you having
> missed that step.
>
> You should see "Calculating new tokens" in the logs of the existing hosts
> if you performed that step correctly, and "nodetool ring" should show that
> the existing hosts each have 256 tokens which are contiguous in the ring.
>
> If you missed this step then the new node will be taking 256 tokens in a
> ring with only N tokens (1 per existing host) and so will end up with
> 256/(256+N) of the data (almost all of it).
>
>
>
> On 28 April 2013 23:01, John Watson <john@disqus.com> wrote:
>
>> On Sun, Apr 28, 2013 at 2:19 PM, aaron morton <aaron@thelastpickle.com>wrote:
>>
>>>  We're going to try running a shuffle before adding a new node again...
>>>> maybe that will help
>>>>
>>> I don't think  hurt but I doubt it will help.
>>>
>>
>> We had to bail on shuffle since we need to add capacity ASAP and not in
>> 20 days.
>>
>>
>>>
>>>    It seems when new nodes join, they are streamed *all* sstables in
>>>>> the cluster.
>>>>>
>>>>>
>>>>>
>>>>> How many nodes did you join, what was the num_tokens ?
>>> Did you notice streaming from all nodes (in the logs) or are you saying
>>> this in response to the cluster load increasing ?
>>>
>>>
>> Was only adding 2 nodes at the time (planning to add a total of 12.)
>> Starting with a cluster of 12, but now 11 since 1 node entered some weird
>> state when one of the new nodes ran out disk space.
>> num_tokens is set to 256 on all nodes.
>> Yes, nearly all current nodes were streaming to the new ones (which was
>> great until disk space was an issue.)
>>
>>>     The purple line machine, I just stopped the joining process because
>>>>> the main cluster was dropping mutation messages at this point on a few
>>>>> nodes (and it still had dozens of sstables to stream.)
>>>>>
>>>>> Which were the new nodes ?
>>> Can you show the output from nodetool status?
>>>
>>>
>> The new nodes are the purple and gray lines above all the others.
>>
>> nodetool status doesn't show joining nodes. I think I saw a bug already
>> filed for this but I can't seem to find it.
>>
>>
>>>
>>> Cheers
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Cassandra Consultant
>>> New Zealand
>>>
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 27/04/2013, at 9:35 AM, Bryan Talbot <btalbot@aeriagames.com> wrote:
>>>
>>> I believe that "nodetool rebuild" is used to add a new datacenter, not
>>> just a new host to an existing cluster.  Is that what you ran to add the
>>> node?
>>>
>>> -Bryan
>>>
>>>
>>>
>>> On Fri, Apr 26, 2013 at 1:27 PM, John Watson <john@disqus.com> wrote:
>>>
>>>> Small relief we're not the only ones that had this issue.
>>>>
>>>> We're going to try running a shuffle before adding a new node again...
>>>> maybe that will help
>>>>
>>>> - John
>>>>
>>>>
>>>> On Fri, Apr 26, 2013 at 5:07 AM, Francisco Nogueira Calmon Sobral <
>>>> fsobral@igcorp.com.br> wrote:
>>>>
>>>>> I am using the same version and observed something similar.
>>>>>
>>>>> I've added a new node, but the instructions from Datastax did not work
>>>>> for me. Then I ran "nodetool rebuild" on the new node. After finished
this
>>>>> command, it contained two times the load of the other nodes. Even when
I
>>>>> ran "nodetool cleanup" on the older nodes, the situation was the same.
>>>>>
>>>>> The problem only seemed to disappear when "nodetool repair" was
>>>>> applied to all nodes.
>>>>>
>>>>> Regards,
>>>>> Francisco Sobral.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Apr 25, 2013, at 4:57 PM, John Watson <john@disqus.com> wrote:
>>>>>
>>>>> After finally upgrading to 1.2.3 from 1.1.9, enabling vnodes, and
>>>>> running upgradesstables, I figured it would be safe to start adding nodes
>>>>> to the cluster. Guess not?
>>>>>
>>>>> It seems when new nodes join, they are streamed *all* sstables in the
>>>>> cluster.
>>>>>
>>>>>
>>>>> https://dl.dropbox.com/s/bampemkvlfck2dt/Screen%20Shot%202013-04-25%20at%2012.35.24%20PM.png
>>>>>
>>>>> The gray the line machine ran out disk space and for some reason
>>>>> cascaded into errors in the cluster about 'no host id' when trying to
store
>>>>> hints for it (even though it hadn't joined yet).
>>>>> The purple line machine, I just stopped the joining process because
>>>>> the main cluster was dropping mutation messages at this point on a few
>>>>> nodes (and it still had dozens of sstables to stream.)
>>>>>
>>>>> I followed this:
>>>>> http://www.datastax.com/docs/1.2/operations/add_replace_nodes
>>>>>
>>>>> Is there something missing in that documentation?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> John
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
> --
> Sam Overton
> Acunu | http://www.acunu.com | @acunu
>

Mime
View raw message