incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adi <adi.pan...@gmail.com>
Subject Re: Bringing a new node to replace a dead node - streaming is very slow
Date Thu, 04 Nov 2010 21:19:43 GMT
> Hello,
>           Environment - Cassandra 0.6.5 CentOS55 amazon ec2 image small.
>
> I am bringing in a new node to replace a dead node. Using new IP and
> deadNodeToken -1 as the InitialToken for the new node. The node has started
> bootstrapping and I can see the seed node is streaming to the new node.
>
> [root@ip-10-122-178-221 apache-cassandra-0.6.5]# bin/nodetool -h
> seed_node streams
> Mode: Normal
> Streaming to: /new_node_ip
>    /data/var/lib/cassandra/data/usertable/stream/data-138-Index.db
> 0/7604053
>    /data/var/lib/cassandra/data/usertable/stream/data-138-Filter.db
> 0/393205
>    /data/var/lib/cassandra/data/usertable/stream/data-138-Data.db
> 0/2527290694
> Not receiving any streams.
>
> But the network IO shown using tcptrack shows just 10-11 KB/s of traffic.
> 1) Is there a way of speeding up this bootstrap process or some other
> recommended way of replacing a dead node.
> 2) Is there a way to find the files a node should be having(say the ones
> that show up in stream command) and just scp them to the new node.
>
> Thank you for your time.
>
> -Adi
>


FYI Ended up re-starting the whole cluster which in effect decommissioned
the dead node and redistributed the data. And then streamed back to the new
node on bootstrap. Saw much better speeds at bootstrap (10-12 MB/s)

For others seeing slow streaming speeds when adding a new replacement node:
Another thing worth trying is re-starting the seed node and then adding the
new node.

-Adi

Mime
View raw message