cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reid Pinchback <rpinchb...@tripadvisor.com>
Subject Re: Cassandra 2.1.18 - Question on stream/bootstrap throughput
Date Tue, 22 Oct 2019 14:31:25 GMT
A high level of compaction seems highly likely to throttle you by sending the service into
a GC death spiral, doubly-so if any repairs happen to be underway at the same time (I may
or may not have killed a few nodes this way, but I admit nothing!).  Even if not in GC hell,
it can cause you to episodically blast out writes that rapidly dirty a lot of pages, thus
triggering a fill of the disk io queue that then starves out read requests from the disk.
 More != Better when it comes to compaction.  You want as little compaction as your usage
pattern requires of you.  Smoothness of its contribution to the overall load is a better objective.

Jon Haddad did a datastax conference talk this year on some easy tunings that you’ll likely
want to listen to. You’ll probably end up rethinking your vnode count as well. Also note
that a fast disk can spend a lot of its time doing the wrong things. His talk covers some
of the factors in that.

https://www.youtube.com/watch?v=swL7bCnolkU


From: "Steinmaurer, Thomas" <thomas.steinmaurer@dynatrace.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Tuesday, October 22, 2019 at 6:47 AM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Cassandra 2.1.18 - Question on stream/bootstrap throughput

Message from External Sender
Hello,

using 2.1.8, 3 nodes (m4.10xlarge, ESB SSD-based), vnodes=256, RF=3, we are trying to add
a 4th node.

The two options to my knowledge, mainly affecting throughput, namely stream output and compaction
throttling has been set to very high values (e.g. stream output = 800 Mbit/s resp. compaction
throughput = 500 Mbyte/s) or even set to 0 (unthrottled) in cassandra.yaml + process restart.
In both scenarios (throttling with high values vs. unthrottled), the 4th node is streaming
from one node capped ~ 180-200Mbit/s, according to our SFM.

The nodes have plenty of resources available (10Gbit, disk io/iops), also confirmed by e.g.
iperf in regard to NW throughput and write to / read from disk in the area of 200 MByte/s.

Are there any other known throughput / bootstrap limitations, which basically outrule above
settings?

Thanks,
Thomas


The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
Mime
View raw message