Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 88693 invoked from network); 29 Mar 2011 22:00:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 29 Mar 2011 22:00:49 -0000 Received: (qmail 99478 invoked by uid 500); 29 Mar 2011 22:00:45 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 99455 invoked by uid 500); 29 Mar 2011 22:00:45 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 99447 invoked by uid 99); 29 Mar 2011 22:00:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 22:00:45 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a48.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 22:00:38 +0000 Received: from homiemail-a48.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a48.g.dreamhost.com (Postfix) with ESMTP id C5FD14F8070 for ; Tue, 29 Mar 2011 15:00:16 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=m8uuZ3vTIr o+uQtUsHuCsz4RM2JwyW1cjoUQ4tFjc/3efhsYJfmjHRsz+bnjaybQtfU2T9p1AM LcP9FLNBirxoRsDJKaYrOxMn1h/k31LQPFN7+QXlpmN0dT9og8WNpEBl2l3lccwT h2uVlohTmNFdI+FJhIOCyRA468hcwDibg= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=XwLLKVWWf/luWz7b McRzQKh1ncM=; b=NfGpzKxSblHbTYd5WWxdPnlk9gSgNP80N4t6e8BDGePxhnnu ran+vEvNKvLqE0tIVSeOP0csge1OB4wCzzpaya30a3ag5luQcPFiZ9bqgzFyqB28 DoHkwjiX95yDstzyfRyoNz5OAVRFFBZPOHBu7NPmzydTfL1RDHeGVABS8oM= Received: from [10.0.1.2] (CPE-58-168-9-85.lns3.cht.bigpond.net.au [58.168.9.85]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a48.g.dreamhost.com (Postfix) with ESMTPSA id 27B4D4F8065 for ; Tue, 29 Mar 2011 15:00:13 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1082.1) Content-Type: multipart/alternative; boundary=Apple-Mail-28-1006263015 Subject: Re: Help on how to configure an off-site DR node. Date: Wed, 30 Mar 2011 09:00:10 +1100 In-Reply-To: <1301405897.12085.26.camel@brian-desktop> To: user@cassandra.apache.org References: <1301312833.14296.14.camel@brian-desktop> <144A7397-276F-4F63-8B93-48EB4154E040@thelastpickle.com> <1301405897.12085.26.camel@brian-desktop> Message-Id: <96943CD0-48C6-4ED3-8D2B-130E1B11BADA@thelastpickle.com> X-Mailer: Apple Mail (2.1082.1) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-28-1006263015 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Snapshots take use a hard link and do not take additional disk space = http://www.mail-archive.com/user@cassandra.apache.org/msg11028.html WRT losing a node, it's not the number of total nodes thats important is = the number of replicas. If you have 3 nodes with RF2 and you lose one of = the replicas you will not be able to work at Quorum level.=20 You *may* be able to use the NetworkTopologySnitch to have 2 replicas in = DC1 and 1 replica in DC2. Then use the property file snitch to only put = one node in the second DC. Finally work against DC1 with LOCAL_QUORUM so = you do not wait on DC 2 and you can tolerate the link to DC2 failing. = That also means there is no guarantee DC2 is up to date. If you were to = ship snapshots you would have a better idea of what you had in DC2.=20 FWIW I'm not convinced that setting things up so that one node gets = *all* the data in DC2 is a good idea. It would make an offsite replica = that could only work at essentially CL ONE and would require a lot of = streaming to move to a cluster with more nodes. I don't have time right = now to think through all of the implications now (may be able to do some = more thinking tonight), but the data stax guide creates a warm fail over = that is ready to work. I'm not sure what this approach would give you in = case of failure: a backup to be restored or a failover installation.=20 Hope that helps.=20 Aaron On 30 Mar 2011, at 00:38, Brian Lycett wrote: > Hi. >=20 > Cheers for your reply. >=20 > Unfortunately there's too much data for snapshots to be practical. = The > data set will be at least 400GB initially, and the offsite node will = be > on a 20Mbit leased line. >=20 > However I don't need the consistency level to be quorum for = read/writes > in the production cluster, so am I right in still assuming that a > replication factor of 2 in a three node cluster allows for one node to > die without data loss? >=20 > If that's the case, I still don't understand how to ensure that the > offsite node will get a copy of the whole data set. > I've read through the O'Reilly book, and that doesn't seem to address > this scenario (unless I still don't get the Cassandra basics at a > fundamental level). >=20 > Does anyone know any tutorials/examples of such a set-up that would = help > me out? >=20 > Cheers, >=20 > Brian >=20 >=20 >=20 > On Tue, 2011-03-29 at 21:56 +1100, aaron morton wrote: >> Be aware that at RF 2 the Quorum is 2, so you cannot afford to lose a >> replica when working at Quorum. 3 is really the starting point if you >> want some redundancy.=20 >>=20 >>=20 >> If you want to get your data offsite how about doing snapshots and >> moving them off >> site http://wiki.apache.org/cassandra/Operations#Consistent_backups >>=20 >>=20 >> The guide from Data Stax will give you a warm failover site, which >> sounds a bit more than what you need. =20 >>=20 >>=20 >> Hope that helps.=20 >> Aaron >>=20 >>=20 >> On 28 Mar 2011, at 22:47, Brian Lycett wrote: >>=20 >>> Hello. >>>=20 >>> I'm setting up a cluster that has three nodes in our production >>> rack. >>> My intention is to have a replication factor of two for this. >>> For disaster recovery purposes, I need to have another node (or >>> two?) >>> off-site. >>>=20 >>> The off-site node is entirely for the purpose of having an offsite >>> backup of the data - no clients will connect to it. >>>=20 >>> My question is, is it possible to configure Cassandra so that the >>> offsite node will have a full copy of the data set? >>> That is, somehow guarantee that a replica of all data will be >>> written to >>> it, but without having to resort to an ALL consistency level for >>> writes? >>> Although the offsite node will on a 20Mbit leased line, I'd rather >>> not >>> have the risk that the link goes down and breaks the cluster. >>>=20 >>> I've seen this suggestion here: >>> http://www.datastax.com/docs/0.7/operations/datacenter#disaster >>> but that configuration is vulnerable to the link breaking, and uses >>> four >>> nodes in the offsite location. >>>=20 >>>=20 >>> Regards, >>>=20 >>> Brian >>>=20 >>>=20 >>>=20 >>=20 >>=20 >=20 >=20 --Apple-Mail-28-1006263015 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii http://www.mail-archive.com/user@cassandra.apache.org/msg11028.html=

WRT losing a node, it's not the number of total nodes thats = important is the number of replicas. If you have 3 nodes with RF2 and = you lose one of the replicas you will not be able to work at Quorum = level. 

You *may* be able to use the = NetworkTopologySnitch to have 2 replicas in DC1 and 1 replica in DC2. = Then use the property file snitch to only put one node in the second DC. = Finally work against DC1 with LOCAL_QUORUM so you do not wait on DC 2 = and you can tolerate the link to DC2 failing. That also means there is = no guarantee DC2 is up to date. If you were to ship snapshots you would = have a better idea of what you had in = DC2. 

FWIW I'm not convinced that setting = things up so that one node gets *all* the data in DC2 is a good idea. It = would make an offsite replica that could only work at essentially CL ONE = and would require a lot of streaming to move to a cluster with more = nodes. I don't have time right now to think through all of the = implications now (may be able to do some more thinking tonight), but the = data stax guide creates a warm fail over that is ready to work. I'm not = sure what this approach would give you in case of failure: a backup to = be restored or a failover = installation. 

Hope that = helps. 
Aaron


=
On 30 Mar 2011, at 00:38, Brian Lycett wrote:

Hi.

Cheers for your = reply.

Unfortunately there's too much data for snapshots to be = practical.  The
data set will be at least 400GB initially, and = the offsite node will be
on a 20Mbit leased line.

However I = don't need the consistency level to be quorum for read/writes
in the = production cluster, so am I right in still assuming that = a
replication factor of 2 in a three node cluster allows for one node = to
die without data loss?

If that's the case, I still don't = understand how to ensure that the
offsite node will get a copy of the = whole data set.
I've read through the O'Reilly book, and that doesn't = seem to address
this scenario (unless I still don't get the Cassandra = basics at a
fundamental level).

Does anyone know any = tutorials/examples of such a set-up that would help
me = out?

Cheers,

Brian



On Tue, 2011-03-29 at = 21:56 +1100, aaron morton wrote:
Be aware = that at RF 2 the Quorum is 2, so you cannot afford to lose = a
replica when working at = Quorum. 3 is really the starting point if = you
want some redundancy. =


If you want to = get your data offsite how about doing snapshots = and
moving them = off
site ht= tp://wiki.apache.org/cassandra/Operations#Consistent_backups


The guide from = Data Stax will give you a warm failover site, = which
sounds a bit more than = what you need.  


Hope that = helps.
Aaron


On 28 Mar 2011, = at 22:47, Brian Lycett wrote:

Hello.

I'm setting up a cluster that = has three nodes in our = production
rack.
My intention is to have a = replication factor of two for = this.
For disaster recovery purposes, I need to have another = node (or
two?)
off-site.

The off-site node is entirely = for the purpose of having an = offsite
backup of the data - no clients will connect to = it.

My question is, is it possible = to configure Cassandra so that = the
offsite node will have a full copy of the data = set?
That is, somehow guarantee that a replica of all data will = be
written to
it, but without having to resort = to an ALL consistency level for
writes?
Although the offsite node will = on a 20Mbit leased line, I'd = rather
not
have the risk that the link goes = down and breaks the cluster.

I've seen this suggestion = here:
h= ttp://www.datastax.com/docs/0.7/operations/datacenter#disaster
but that configuration is vulnerable to the link breaking, = and uses
four
nodes in the offsite = location.


Regards,

Brian








= = --Apple-Mail-28-1006263015--