incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Cowen <d...@luciddg.com>
Subject nodetool repair fails after expansion
Date Fri, 04 Oct 2013 21:08:06 GMT
We're testing expanding a 4-node cluster into an 8-node cluster, and we
keep running into issues with the repair process near the end.

We're bringing up nodes 1-by-1 into the cluster, retokening nodes for an
8-node configuration, running nodetool cleanup on the nodes after each
retokening, and then increasing the replication factor to 5. This all works
without issue, and the cluster appears to be healthy in that 8-node
configuration with a replication factor of 5.

However, when we then run nodetool repair on the nodes, it will at some
point stall, even when being run on one of the new nodes.

It doesn't appear to stall while it's performing a compaction or
transferring CF data. We've monitored compactionstats and netstats closely,
and things always stall when a repair command is started, ie:

[2013-10-02 23:19:39,254] Starting repair command #9, repairing 5 ranges
for keyspace ourkeyspace

The last message from AntiEntropyService is usually something to the effect
of:

<190>Oct  3 00:01:02 myhost.com 1970947950 [AntiEntropySessions:24] INFO
 org.apache.cassandra.service.AntiEntropyService  - [repair
#9b17d310-2bbd-11e3-0000-e06ec6c436ff] session completed successfully

... and then things don't start for the next repair. Nothing in the logs
that looks related.

Where this occurs is arbitrary. If I run on individual CFs within
ourkeyspace, some will succeed, and some will fail, but if we start over
and do the 4-node to 8-node expansion again, things will fail at a
different place.

Advice as to what to look at next?

Thanks,

Dave

Mime
View raw message