incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ZFabrik Subscriber <subscri...@zfabrik.de>
Subject Re: How to scale Cassandra?
Date Mon, 04 Jul 2011 20:13:33 GMT
Hi SC, 

I'm just talking about workload in general. The point is that sooner or later you come to
the point that you need to scale-out. And the question is, what's the best strategy here?
Especially when your cluster is almost balanced.

500 GB seems to be a good ball-park figure, I think I read this number somewhere else recently.

Regards
Udo

Am 04.07.2011 um 21:10 schrieb Sebastien Coutu:

> Hi Udo,
> 
> I didn't read the whole thread but can you define the type of workload you're looking
at? Do you have jobs that require reading the whole data stored in your database? For example
one big column family that needs to be read entirely by a job? Because the amount of time
required to read a whole disk (SATA II) of 1TB is roughly 2-2.5 hours. Now add RAID to this
and you can modify this amount of time but your bottleneck will pretty much always be your
disks. On our cluster, we currently have more than 1TB per node and it holds but we find that
our sweet spot should be around 400-500GB per node.
> 
> Regards,
> 
> SC
> 
> 
> On Mon, Jul 4, 2011 at 3:01 PM, ZFabrik Subscriber <subscriber@zfabrik.de> wrote:
> Let's assume you have 50 nodes and their work-load grows simultaneously. You discover
that the nodes are about to reach their limits (btw. what is the actual limit of a Cassandra
node? 100GB? 500GB? 1TB?) 
> You decide to add another 50 nodes. Do you do this within one step? Or one after the
other? Or in several rounds, always every RF-rd node?
> Or you add 20 nodes and move the token ranges. Again, in one step? 20 steps? 4 steps
5 nodes each?
> This could take a while (in terms of days, if not weeks) in larger clusters!
> 
> Does anybody has experience with real life scale-outs?
> 
> Regards
> Udo
> 
> Am 04.07.2011 um 16:21 schrieb Paul Loy:
> 
>> Well, by issuing a nodetool move when a node is under high load, you basically make
that node unresponsive. That's fine, but a nodetool move on one node also means that that
node's replica data needs to move around the ring and possibly some replica data from the
next (or previous) node in the ring. So how does this affect other nodes wrt RF and quorum?
Will quorum fail until the replicas have moved also?
>> 
>> On Mon, Jul 4, 2011 at 3:08 PM, Dan Hendry <dan.hendry.junk@gmail.com> wrote:
>> Moving nodes does not result in downtime provide you use proper replication factors
and read/write consistencies. The typical recommendation is RF=3 and QUORUM reads/writes.
>> 
>>  
>> 
>> Dan
>> 
>>  
>> 
>> From: Paul Loy [mailto:keteracel@gmail.com] 
>> Sent: July-04-11 5:59
>> To: user@cassandra.apache.org
>> Subject: Re: How to scale Cassandra?
>> 
>>  
>> 
>> That's basically how I understand it.
>> 
>> However, I think it gets better with larger clusters as the proportion of the ring
you move around at any time is much lower.
>> 
>> On Mon, Jul 4, 2011 at 10:54 AM, Subscriber <subscriber@zfabrik.de> wrote:
>> 
>> Hi there,
>> 
>> I read a lot of Cassandra's high scalability feature: allowing seamless addition
of nodes, no downtime etc.
>> But I wonder how one will do this in practice in an operational system.
>> 
>> In the system we're going to implement we're expecting a huge number of writes with
uniformly distributed keys
>> (the keys are given and cannot be generated). That means using RandomPartitioner
will (more or less) result in
>> the same work-load per node as any other OrderPreservePartitioner - right?
>> 
>> But how do you scale a (more or less) balanced Cassandra cluster? I think that in
the end
>> you always have to double the number of nodes (adding just a handful of nodes disburdens
only the split regions, the
>> work-load of untouched regions will grow with unchanged speed).
>> 
>> This seems to be ok for small clusters. But what do you do with when you have several
100s of nodes in your cluster?
>> It seems to me that a balanced cluster is a bless for performance but a curse for
scalability...
>> 
>> What are the alternatives? One could re-distribute the token ranges, but this would
cause
>> downtimes (AFAIK); not an option!
>> 
>> Is there anything that I didn't understand or do I miss something else? Is the only
left strategy to make sure that
>> the cluster grows unbalanced so one can add nodes to the hotspots? However in this
case you have to make sure
>> that this strategy is lasting. Could be too optimistic...
>> 
>> Best Regards
>> Udo
>> 
>> 
>> 
>> 
>> -- 
>> ---------------------------------------------
>> Paul Loy
>> paul@keteracel.com
>> http://uk.linkedin.com/in/paulloy
>> 
>> No virus found in this incoming message.
>> Checked by AVG - www.avg.com
>> Version: 9.0.901 / Virus Database: 271.1.1/3743 - Release Date: 07/04/11 02:35:00
>> 
>> 
>> 
>> 
>> -- 
>> ---------------------------------------------
>> Paul Loy
>> paul@keteracel.com
>> http://uk.linkedin.com/in/paulloy
> 
> 


Mime
View raw message