I am trying to rebuild a new Data Center with 50 Nodes, and expect 1 TB / node. Nodes are backed by SSDs, and the rebuild is happening from another DC in same physical region. This is with 2.1.13.
I am doing this with stream_throughput=200 MB, concurrent_compactors=256, compactionthroughput=0, and memtable_cleanup_threshold=0.
7. (memtable setting was necessary to keep # SSTable files in check) and running rebuild 20 nodes at a time.
Have people generally attempted to do such large rebuilds ? Any tips ?