So we are adding another node to the cluster with the latest 0.6 branch (RC1).  It seems to be hung in some limbo state.

Before bootstrapping our cluster had 50-60GB spread fairly evenly across 4 machines, with RF=3.   One machine had more load than the others, and sure enough bootstrapping selected that node.   That is the red machine.  The light blue machine is the new machine.

I have attached a graph to illustrate when the bootstrap process started.

In jconsole the streamingservice status was "performing anticompaction..." for over 18-24 hrs.  It is currently in "nothing is happening".   It did have 1 active STREAM-STAGE task, but the machine had to be rebooted for something unrelated to cassandra. Now the light blue machine appears to be getting data, but its growing at virtually the same rate as the other machines which makes me think it is part of the cluster and not actually streaming data from the machine its supposed to.  

Any other ideas on how to debug?


--
Dan Di Spaltro