Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C5E4911DCC for ; Wed, 21 May 2014 17:09:02 +0000 (UTC) Received: (qmail 12474 invoked by uid 500); 21 May 2014 17:09:00 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 12439 invoked by uid 500); 21 May 2014 17:09:00 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 12431 invoked by uid 99); 21 May 2014 17:09:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 May 2014 17:09:00 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ben@instaclustr.com designates 209.85.192.176 as permitted sender) Received: from [209.85.192.176] (HELO mail-pd0-f176.google.com) (209.85.192.176) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 May 2014 17:08:56 +0000 Received: by mail-pd0-f176.google.com with SMTP id p10so1572988pdj.21 for ; Wed, 21 May 2014 10:08:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=k+1FmkTTmcqCq2wJCcSbsnShZXUamYXtMFDrDRx2p5k=; b=a9qzk+KJ/GCHIwZOZlykotYBKccdg1QRREODVVa4QEsktBnqaG531blsXxeBZvpm1j HQPy5uoDGIOBpzRWFyrIBiePE8kXbw3quAmwKZDt5sNLBJKPZWkWMnKHM0Sv/VaQSXgS uXUSVMQPj/t+uVLNTYxrKMHp5vc96v/pq03Sn4SsBasQAbKrmUfQNFpIayzBJlW9TKUl O5yqicoTcDnxFqpF0UgRXv/RUPloCGgEkidNslkn3caZDuA3nUZilwqZKKjR6mmce4zA LzHfYkhkxq2vt4gKda5VruFv23cWLwsFnL6XJjMQ23hNPaBA6tK+9OhO6dT79PFpD3wO 1ctg== X-Gm-Message-State: ALoCoQk5IOgBht7y+BSK8Hi0p/b+9tb27VHTgKWwfORIMTpROhAkKewNZKV7+N/3K0ag5qnHPbae X-Received: by 10.68.178.194 with SMTP id da2mr59905980pbc.151.1400692115583; Wed, 21 May 2014 10:08:35 -0700 (PDT) Received: from [172.31.57.230] ([216.9.110.15]) by mx.google.com with ESMTPSA id bu1sm8999495pbb.54.2014.05.21.10.08.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 21 May 2014 10:08:34 -0700 (PDT) From: Ben Bromhead Content-Type: multipart/alternative; boundary="Apple-Mail=_352A35A3-A87A-402B-8D85-D588019E7093" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: autoscaling cassandra cluster Date: Wed, 21 May 2014 10:08:34 -0700 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1508) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_352A35A3-A87A-402B-8D85-D588019E7093 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 The mechanics for it are simple compared to figuring out when to scale, = especially when you want to be scaling before peak load on your cluster = (adding and removing nodes puts additional load on your cluster). We are currently building our own in-house solution for this for our = customers. If you want to have a go at it yourself, this is a good = starting point: = http://techblog.netflix.com/2013/11/scryer-netflixs-predictive-auto-scalin= g.html = http://techblog.netflix.com/2013/12/scryer-netflixs-predictive-auto-scalin= g.html Most of this is fairly specific to Netflix, but an interesting read = nonetheless. Datastax OpsCenter also provides capacity planning and forecasting and = can provide an easy set of metrics you can make your scaling decisions = on. = http://www.datastax.com/what-we-offer/products-services/datastax-opscenter= =20 Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359 On 21/05/2014, at 7:51 AM, James Horey wrote: > If you're interested and/or need some Cassandra docker images let me = know I'll shoot you a link. >=20 > James >=20 > Sent from my iPhone >=20 > On May 21, 2014, at 10:19 AM, Jabbar Azam wrote: >=20 >> That sounds interesting. I was thinking of using coreos with docker = containers for the business logic, frontend and Cassandra. I'll also = have a look at cassandra-mesos >>=20 >> Thanks >>=20 >> Jabbar Azam >>=20 >> On 21 May 2014 14:04, "Panagiotis Garefalakis" = wrote: >> I agree with Prem, but recently a guy send this promising project = called Mesos in this list.=20 >> https://github.com/mesosphere/cassandra-mesos >> One of its goals is to make scaling easier.=20 >> I don=92t have any personal opinion yet but maybe you could give it a = try. >>=20 >> Regards, >> Panagiotis >>=20 >>=20 >>=20 >> On Wed, May 21, 2014 at 3:49 PM, Jabbar Azam = wrote: >> Hello Prem, >>=20 >> I'm trying to find out whether people are autoscaling up and down = automatically, not manually. I'm also interested in whether they are = using a cloud based solution and creating and destroying instances.=20 >>=20 >> I've found the following regarding GCE = https://cloud.google.com/developers/articles/auto-scaling-on-the-google-cl= oud-platform and how instances can be created and destroyed.=20 >>=20 >> I >>=20 >>=20 >> Thanks >>=20 >> Jabbar Azam >>=20 >>=20 >> On 21 May 2014 13:09, Prem Yadav wrote: >> Hi Jabbar, >> with vnodes, scaling up should not be a problem. You could just add a = machines with the cluster/seed/datacenter conf and it should join the = cluster. >> Scaling down has to be manual where you drain the node and = decommission it. >>=20 >> thanks, >> Prem >>=20 >>=20 >>=20 >> On Wed, May 21, 2014 at 12:35 PM, Jabbar Azam = wrote: >> Hello, >>=20 >> Has anybody got a cassandra cluster which autoscales depending on = load or times of the day? >>=20 >> I've seen the documentation on the datastax website and that only = mentioned adding and removing nodes, unless I've missed something. >>=20 >> I want to know how to do this for the google compute engine. This = isn't for a production system but a test system(multiple nodes) where I = want to learn. I'm not sure how to check the performance of the cluster, = whether I use one performance metric or a mix of performance metrics and = then invoke a script to add or remove nodes from the cluster. >>=20 >> I'd be interested to know whether people out there are autoscaling = cassandra on demand. >>=20 >> Thanks >>=20 >> Jabbar Azam >>=20 >>=20 >>=20 --Apple-Mail=_352A35A3-A87A-402B-8D85-D588019E7093 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 The = mechanics for it are simple compared to figuring out when to scale, = especially when you want to be scaling before peak load on your cluster = (adding and removing nodes puts additional load on your = cluster).

We are currently building our own in-house = solution for this for our customers. If you want to have a go at it = yourself, this is a good starting point:


Most of this is = fairly specific to Netflix, but an interesting read = nonetheless.

Datastax OpsCenter also provides = capacity planning and forecasting and can provide an easy set of metrics = you can make your scaling decisions on.


www.instaclustr.com | = @instaclustr | +61 = 415 936 359




On 21/05/2014, at 7:51 AM, James Horey <jlh@opencore.io> wrote:

If you're interested and/or need some Cassandra docker = images let me know I'll shoot you a = link.

James

Sent from my = iPhone

On May 21, 2014, at 10:19 AM, Jabbar Azam <ajazam@gmail.com> = wrote:

That sounds = interesting.   I was thinking of using coreos with docker = containers for the business logic, frontend and Cassandra. I'll also = have a look at cassandra-mesos

Thanks

Jabbar Azam

On 21 May 2014 14:04, "Panagiotis = Garefalakis" <pangaref@gmail.com> wrote:
I agree with Prem, but recently a guy = send this promising project called Mesos in this list.
https://github.com/mesosphere/cassandra-mesos
One of its goals is to make scaling easier.
I don=92t have any = personal opinion yet but maybe you could give it a = try.

Regards,
Panagiotis



On Wed, May 21, 2014 at 3:49 PM, = Jabbar Azam <ajazam@gmail.com> wrote:
Hello Prem,

I'm trying to find out = whether people are autoscaling up and down automatically, not manually. = I'm also interested in whether they are using a cloud based solution and = creating and destroying instances. 

I've found the following regarding GCE https://cloud.google.com/developers/articles/auto-scalin= g-on-the-google-cloud-platform and how instances can be created and = destroyed. 

 I


Thanks

Jabbar = Azam


On 21 May 2014 13:09, Prem Yadav = <ipremyadav@gmail.com> = wrote:
Hi Jabbar,
with vnodes, scaling up should not be a = problem. You could just add a machines with the cluster/seed/datacenter = conf and it should join the cluster.
Scaling down has to be = manual where you drain the node and decommission it.
=

thanks,
Prem

<= div>


On = Wed, May 21, 2014 at 12:35 PM, Jabbar Azam <ajazam@gmail.com> wrote:
Hello,

Has anybody got a cassandra = cluster which autoscales depending on load or times of the day?

I've seen the documentation on the datastax website = and that only mentioned adding and removing nodes, unless I've missed = something.

I want to know how to do this for the google compute = engine. This isn't for a production system but a test system(multiple = nodes) where I want to learn. I'm not sure how to check the performance = of the cluster, whether I use one performance metric or a mix of = performance metrics and then invoke a script to add or remove nodes from = the cluster.

I'd be interested to know whether people out there = are autoscaling cassandra on demand.

Thanks

Jabbar Azam




= --Apple-Mail=_352A35A3-A87A-402B-8D85-D588019E7093--