incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Talbot <btal...@aeriagames.com>
Subject Re: Adding nodes in 1.2 with vnodes requires huge disks
Date Fri, 26 Apr 2013 21:35:07 GMT
I believe that "nodetool rebuild" is used to add a new datacenter, not just
a new host to an existing cluster.  Is that what you ran to add the node?

-Bryan



On Fri, Apr 26, 2013 at 1:27 PM, John Watson <john@disqus.com> wrote:

> Small relief we're not the only ones that had this issue.
>
> We're going to try running a shuffle before adding a new node again...
> maybe that will help
>
> - John
>
>
> On Fri, Apr 26, 2013 at 5:07 AM, Francisco Nogueira Calmon Sobral <
> fsobral@igcorp.com.br> wrote:
>
>> I am using the same version and observed something similar.
>>
>> I've added a new node, but the instructions from Datastax did not work
>> for me. Then I ran "nodetool rebuild" on the new node. After finished this
>> command, it contained two times the load of the other nodes. Even when I
>> ran "nodetool cleanup" on the older nodes, the situation was the same.
>>
>> The problem only seemed to disappear when "nodetool repair" was applied
>> to all nodes.
>>
>> Regards,
>> Francisco Sobral.
>>
>>
>>
>>
>> On Apr 25, 2013, at 4:57 PM, John Watson <john@disqus.com> wrote:
>>
>> After finally upgrading to 1.2.3 from 1.1.9, enabling vnodes, and running
>> upgradesstables, I figured it would be safe to start adding nodes to the
>> cluster. Guess not?
>>
>> It seems when new nodes join, they are streamed *all* sstables in the
>> cluster.
>>
>>
>> https://dl.dropbox.com/s/bampemkvlfck2dt/Screen%20Shot%202013-04-25%20at%2012.35.24%20PM.png
>>
>> The gray the line machine ran out disk space and for some reason cascaded
>> into errors in the cluster about 'no host id' when trying to store hints
>> for it (even though it hadn't joined yet).
>> The purple line machine, I just stopped the joining process because the
>> main cluster was dropping mutation messages at this point on a few nodes
>> (and it still had dozens of sstables to stream.)
>>
>> I followed this:
>> http://www.datastax.com/docs/1.2/operations/add_replace_nodes
>>
>> Is there something missing in that documentation?
>>
>> Thanks,
>>
>> John
>>
>>
>>
>

Mime
View raw message