Small relief we're not the only ones that had this issue.We're going to try running a shuffle before adding a new node again... maybe that will help- JohnOn Fri, Apr 26, 2013 at 5:07 AM, Francisco Nogueira Calmon Sobral <email@example.com> wrote:
I am using the same version and observed something similar.I've added a new node, but the instructions from Datastax did not work for me. Then I ran "nodetool rebuild" on the new node. After finished this command, it contained two times the load of the other nodes. Even when I ran "nodetool cleanup" on the older nodes, the situation was the same.The problem only seemed to disappear when "nodetool repair" was applied to all nodes.Regards,Francisco Sobral.
On Apr 25, 2013, at 4:57 PM, John Watson <firstname.lastname@example.org> wrote:After finally upgrading to 1.2.3 from 1.1.9, enabling vnodes, and running upgradesstables, I figured it would be safe to start adding nodes to the cluster. Guess not?It seems when new nodes join, they are streamed *all* sstables in the cluster.The gray the line machine ran out disk space and for some reason cascaded into errors in the cluster about 'no host id' when trying to store hints for it (even though it hadn't joined yet).The purple line machine, I just stopped the joining process because the main cluster was dropping mutation messages at this point on a few nodes (and it still had dozens of sstables to stream.)I followed this: http://www.datastax.com/docs/1.2/operations/add_replace_nodesIs there something missing in that documentation?Thanks,John