cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleksandr Shulgin <oleksandr.shul...@zalando.de>
Subject Re: Cassandra 2.1.18 - Question on stream/bootstrap throughput
Date Tue, 22 Oct 2019 14:35:09 GMT
On Tue, Oct 22, 2019 at 12:47 PM Steinmaurer, Thomas <
thomas.steinmaurer@dynatrace.com> wrote:

>
>
> using 2.1.8, 3 nodes (m4.10xlarge, ESB SSD-based), vnodes=256, RF=3, we
> are trying to add a 4th node.
>
>
>
> The two options to my knowledge, mainly affecting throughput, namely
> stream output and compaction throttling has been set to very high values
> (e.g. stream output = 800 Mbit/s resp. compaction throughput = 500 Mbyte/s)
> or even set to 0 (unthrottled) in cassandra.yaml + process restart. In both
> scenarios (throttling with high values vs. unthrottled), the 4th node is
> streaming from one node capped ~ 180-200Mbit/s, according to our SFM.
>
>
>
> The nodes have plenty of resources available (10Gbit, disk io/iops), also
> confirmed by e.g. iperf in regard to NW throughput and write to / read from
> disk in the area of 200 MByte/s.
>
>
>
> Are there any other known throughput / bootstrap limitations, which
> basically outrule above settings?
>

Hi Thomas,

Assuming you have 3 Availability Zones and you are adding the new node to
one of the zones where you already have a node running, it is expected that
it only streams from that node (its local rack).

Have you increased the streaming throughput on the node it streams from or
only on the new node?  The limit applies to the source node as well.  You
can change it online w/o the need to restart using nodetool command.

Have you checked if the new node is not CPU-bound?  It's unlikely though
due to big instance type and only one node to stream from, more relevant
for scenarios when streaming from a lot of nodes.

Cheers,
--
Alex

Mime
View raw message