cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reid Pinchback <>
Subject Re: merge two cluster
Date Wed, 23 Oct 2019 16:30:41 GMT
I haven’t seen much evidence that larger cluster = more performance, plus or minus the statistics
of speculative retry.  It horizontally scales for storage definitely, and somewhat for connection
volume.  If anything, per Sean’s observation, you have less ability to have a stable tuning
for a particular usage pattern.

Try to have a mental picture of what you think is happening in the JVM while Cassandra is
running.  There are short-lived objects, medium-lived objects, long/static-lived objects,
and behind the scenes some degree of read I/O and write I/O against disk.  Garbage collectors
struggle badly with medium-lived objects, but Cassandra really depends a great deal on those.
 If you merge two clusters together, within any one node you still have the JVM size and disk
architecture you had before, but you are adding competition on fixed resources and potentially
in the very way they find most difficult to handle.

If those resources were heavily underutilized, like Sean’s point about merging small apps
together, then sure.  But if those two clusters of yours are already showing that they experience
significant load, then you are unlikely to improve anything, far more likely to end up worse
off.  GC overhead and compaction flushes to disk are your challenges; merging two clusters
doesn’t change the physics of those two areas, but could increase the demand on them.

The only caveat to all of the above I can think of is if there was a fault-tolerance story
motivating the merging.  Like “management wants us in two AZs in AWS, but lacks the budget
for more instances, and each pool by itself is too small for us to come up with a 2 rack organization
that makes sense”.


Reply-To: "" <>
Date: Wednesday, October 23, 2019 at 10:40 AM
To: "" <>
Subject: Re: merge two cluster

Message from External Sender

Sorry, missing question;

Actually I'm asking this for performance perspective. At application level both cluster used
at the same time and approx same level. Inserted data inserted to both cluster, different
parts of course.

If I merge two cluster, can I gain some performance improvements? Like raid stripes, more
disk, more stripe, more speed..

On 23.10.2019 17:30, Durity, Sean R wrote:
Beneficial to whom? The apps, the admins, the developers?

I suggest that app teams have separate clusters per application. This prevents the noisy neighbor
problem, isolates any security issues, and helps when it is time for maintenance, upgrade,
performance testing, etc. to not have to coordinate multiple app teams at the same time. Also,
an individual cluster can be tuned for its specific workload. Sometimes, though, costs and
data size push us towards combining smaller apps owned by the same team onto a single cluster.
Those are the exceptions.

As a Cassandra admin, I am always trying to scale the ability to admin multiple clusters without
just adding new admins. That is an on-going task, dependent on your operating environment.

Also, because every table has a portion of memory (memtable), there is a practical limit to
the number of tables that any one cluster should have. I have heard it is in the low hundreds
of tables. This puts a limit on the number of applications that a cluster can safely support.

Sean Durity – Staff Systems Engineer, Cassandra

From: Osman YOZGATLIOĞLU <><>
Sent: Wednesday, October 23, 2019 6:23 AM
Subject: [EXTERNAL] merge two cluster


I have two cluster and both contains different data sets with different node counts.

Would it be beneficial to merge two cluster?




The information in this Internet Email is confidential and may be legally privileged. It is
intended solely for the addressee. Access to this Email by anyone else is unauthorized. If
you are not the intended recipient, any disclosure, copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject to the terms and
conditions expressed in any applicable governing The Home Depot terms of business or client
engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy
and content of this attachment and for any damages or losses arising from any inaccuracies,
errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature,
which may be contained in this attachment and shall not be liable for direct, indirect, consequential
or special damages in connection with this e-mail message or its attachment.
View raw message