My obvious concern is that as this application grows and we get more companies using our it we will eventually have too many keyspaces to perform repairs on the cluster.We have an application that has been designed to use potentially 100s of keyspaces (one for each company).Is this the behaviour you would expect?
One thing we are noticing is that nodetool repair across all of the keyspaces seems to increase linearly based on the number of keyspaces. For example, if we have a 6 node ec2 (m1.large) cluster across 3 Availability Zones and create 20 keyspaces a nodetool repair -pr on one node takes 3 hours even with no data in any of the keyspaces. If I bump that up to 40 keyspaces it takes 6 hours.
Is there anything you can think of (short of redesigning the cluster to limit keyspaces) to increase the performance of the nodetool repairs?