Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4DA8911E13 for ; Wed, 11 Jun 2014 14:57:15 +0000 (UTC) Received: (qmail 22795 invoked by uid 500); 11 Jun 2014 14:57:11 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 22754 invoked by uid 500); 11 Jun 2014 14:57:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 22746 invoked by uid 99); 11 Jun 2014 14:57:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Jun 2014 14:57:11 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.216.53] (HELO mail-qa0-f53.google.com) (209.85.216.53) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Jun 2014 14:57:07 +0000 Received: by mail-qa0-f53.google.com with SMTP id j15so651940qaq.40 for ; Wed, 11 Jun 2014 07:56:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=LPnVNeJN0tOk/iQqxVvnDsyK9ZyVuoOUD4FkIo93nWM=; b=F4gHDmGYU7/bb442lHgsEN3F3O7pcbwQZtKrKuYPPh6ugUZSWzFpjr3XV6NOYC94Kv AyU9UNoealWtqlM65ljPNXcLvPlTSYd1/P5g3B6xTEbe8X+8pgZhEsU4CIr2RV3k+lyF gLSt1PFzQ9BDjwc6caEI6QVRgGAsoTTof2mpY2+/wV04UOAIPmH3s8wM2dNw8w5nVu5v Pn4Ej7jW/k2k6c3mL5gJVarfzqCczbzdeAG9bvHQ6QP7Rp7u90uRxGTfaDAMdAYvIykU RudxcLAEdYlR6G3GCiuSrWAMefXfQ94Rur0o8Fm0npzQxGr8wFYqmSWHu2g2Q3Y+FtvJ 4euQ== X-Gm-Message-State: ALoCoQlox7yyujmei4C+jZvSQ59IPjy0wWfaGSYKfWTFnAEIow1ekaRYLq8TFRnLsj2TqAtoS0Ck MIME-Version: 1.0 X-Received: by 10.140.97.227 with SMTP id m90mr5696725qge.15.1402498602850; Wed, 11 Jun 2014 07:56:42 -0700 (PDT) Received: by 10.229.230.194 with HTTP; Wed, 11 Jun 2014 07:56:42 -0700 (PDT) In-Reply-To: References: <1401981267.29077.YahooMailNeo@web121903.mail.ne1.yahoo.com> <1401989875.13156.YahooMailNeo@web121906.mail.ne1.yahoo.com> <1401994290.53645.YahooMailNeo@web121905.mail.ne1.yahoo.com> <3AE458EEB5C81A49B12A7AF898C7CD63F7FD4BFE@USOLACRPMBX01.premconf.com> Date: Wed, 11 Jun 2014 07:56:42 -0700 Message-ID: Subject: Re: VPC AWS From: Peter Sanford To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113a2a2e057c7904fb90a788 X-Virus-Checked: Checked by ClamAV on apache.org --001a113a2a2e057c7904fb90a788 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Tinc's developers acknowledge that there are some fairly serious unfixed security issues in their protocol: http://www.tinc-vpn.org/security/. As such, I do not consider tinc to be a good choice for production systems. Either IPSec or OpenVPN are reasonable for connecting VPCs in different regions, and Amazon has published guides for both methods[1][2]. We use IPSec because we have a lot of experience with it, but I'm hesitant to recommend it because it is easy to configure in an insecure manner. [1]: https://aws.amazon.com/articles/5472675506466066 [2]: https://aws.amazon.com/articles/0639686206802544 On Tue, Jun 10, 2014 at 6:29 PM, Ben Bromhead wrote: > Have a look at http://www.tinc-vpn.org/, mesh based and handles multiple > gateways for the same network in a graceful manner (so you can run two > gateways per region for HA). > > Also supports NAT traversal if you need to do public-private clusters. > > We are currently evaluating it for our managed Cassandra in a VPC > solution, but we haven=E2=80=99t ever used it in a production environment= or with a > heavy load, so caveat emptor. > > As for the snitch=E2=80=A6 the GPFS is definitely the most flexible. > > Ben Bromhead > Instaclustr | www.instaclustr.com | @instaclustr > | +61 415 936 359 > > On 10 Jun 2014, at 1:42 am, Ackerman, Mitchell > wrote: > > Peter, > > I too am working on setting up a multi-region VPC Cassandra cluster. Eac= h > region is connected to each other via an OpenVPN tunnel, so we can use > internal IP addresses for both the seeds and broadcast address. This > allows us to use the EC2Snitch (my interpretation of the caveat that this > snitch won=E2=80=99t work in a multi-region environment is that it won=E2= =80=99t work if > you can=E2=80=99t use internal IP addresses, which we can via the VPN tun= nels). > All the C* nodes find each other, and nodetool (or OpsCenter) shows that > we have established a multi-datacenter cluster. > > Thus far, I=E2=80=99m not happy with the performance of the cluster in su= ch a > configuration, but I don=E2=80=99t think that it is related to this confi= guration, > though it could be. > > Mitchell > > *From:* Peter Sanford [mailto:psanford@retailnext.net > ] > *Sent:* Monday, June 09, 2014 7:19 AM > *To:* user@cassandra.apache.org > *Subject:* Re: VPC AWS > > Your general assessments of the limitations of the Ec2 snitches seem to > match what we've found. We're currently using the > GossipingPropertyFileSnitch in our VPCs. This is also the snitch to use i= f > you ever want to have a DC in EC2 and a DC with another hosting provider. > > -Peter > > > On Mon, Jun 9, 2014 at 5:48 AM, Alain RODRIGUEZ > wrote: > Hi guys, there is a lot of answer, it looks like this subject is > interesting a lot of people, so I will end up letting you know how it wen= t > for us. > > For now, we are still doing some tests. > > Yet I would like to know how we are supposed to configure Cassandra in > this environment : > > - VPC > - Multiple datacenters (should be VPCs, one per region, linked through VP= N > ?) > - Cassandra 1.2 > > We are currently running under EC2MultiRegionSnitch, but with no VPC. Our > VPC will have no public interface, so I am not sure how to configure > broadcast address or seeds that are supposed to be the public IP of the > node. > > I could use EC2Snitch, but will cross region work properly ? > > Should I use an other snitch ? > > Is someone using a similar configuration ? > > Thanks for information already given guys, we will achieve this ;-). > > > 2014-06-07 0:05 GMT+02:00 Jonathan Haddad : > > > This may not help you with the migration, but it may with maintenance & > management. I just put up a blog post on managing VPC security groups wi= th > a tool I open sourced at my previous company. If you're going to have > different VPCs (staging / prod), it might help with managing security > groups. > > http://rustyrazorblade.com/2014/06/an-introduction-to-roadhouse/ > > Semi shameless plug... but relevant. > > > On Thu, Jun 5, 2014 at 12:01 PM, Aiman Parvaiz wrote: > Cool, thanks again for this. > > > On Thu, Jun 5, 2014 at 11:51 AM, Michael Theroux > wrote: > You can have a ring spread across EC2 and the public subnet of a VPC. > That is how we did our migration. In our case, we simply replaced the > existing EC2 node with a new instance in the public VPC, restored from a > backup taken right before the switch. > > -Mike > > ------------------------------ > *From:* Aiman Parvaiz > *To:* Michael Theroux > *Cc:* "user@cassandra.apache.org" > *Sent:* Thursday, June 5, 2014 2:39 PM > *Subject:* Re: VPC AWS > > Thanks for this info Michael. As far as restoring node in public VPC is > concerned I was thinking ( and I might be wrong here) if we can have a ri= ng > spread across EC2 and public subnet of a VPC, this way I can simply > decommission nodes in Ec2 as I gradually introduce new nodes in public > subnet of VPC and I will end up with a ring in public subnet and then > migrate them from public to private in a similar way may be. > > If anyone has any experience/ suggestions with this please share, would > really appreciate it. > > Aiman > > > On Thu, Jun 5, 2014 at 10:37 AM, Michael Theroux > wrote: > The implementation of moving from EC2 to a VPC was a bit of a juggling > act. Our motivation was two fold: > > 1) We were running out of static IP addresses, and it was becoming > increasingly difficult in EC2 to design around limiting the number of > static IP addresses to the number of public IP addresses EC2 allowed > 2) VPC affords us an additional level of security that was desirable. > > However, we needed to consider the following limitations: > > 1) By default, you have a limited number of available public IPs for both > EC2 and VPC. > 2) AWS security groups need to be configured to allow traffic for > Cassandra to/from instances in EC2 and the VPC. > > You are correct at the high level that the migration goes from EC2->Publi= c > VPC (VPC with an Internet Gateway)->Private VPC (VPC with a NAT). The > first phase was moving instances to the public VPC, setting broadcast and > seeds to the public IPs we had available. Basically: > > 1) Take down a node, taking a snapshot for a backup > 2) Restore the node on the public VPC, assigning it to the correct > security group, manually setting the seeds to other available nodes > 3) Verify the cluster can communicate > 4) Repeat > > Realize the NAT instance on the private subnet will also require a public > IP. What got really interesting is that near the end of the process we r= an > out of available IPs, requiring us to switch the final node that was on E= C2 > directly to the private VPC (and taking down two nodes at once, which our > setup allowed given we had 6 nodes with an RF of 3). > > What we did, and highly suggest for the switch, is to write down every > step that has to happen on every node during the switch. In our case, ma= ny > of the moved nodes required slightly different configurations for items > like the seeds. > > Its been a couple of years, so my memory on this maybe a little fuzzy :) > > -Mike > > ------------------------------ > *From:* Aiman Parvaiz > *To:* user@cassandra.apache.org; Michael Theroux > *Sent:* Thursday, June 5, 2014 12:55 PM > *Subject:* Re: VPC AWS > > Michael, > Thanks for the response, I am about to head in to something very similar > if not exactly same. I envision things happening on the same lines as you > mentioned. > I would be grateful if you could please throw some more light on how you > went about switching cassandra nodes from public subnet to private with o= ut > any downtime. > I have not started on this project yet, still in my research phase. I pla= n > to have a ec2+public VPC cluster and then decomission ec2 nodes to have > everything in public subnet, next would be to move it to private subnet. > > Thanks > > > On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux > wrote: > We personally use the EC2Snitch, however, we don't have the multi-region > requirements you do, > > -Mike > > ------------------------------ > *From:* Alain RODRIGUEZ > *To:* user@cassandra.apache.org > *Sent:* Thursday, June 5, 2014 9:14 AM > *Subject:* Re: VPC AWS > > I think you can define VPC subnet to be public (to have public + private > IPs) or private only. > > Any insight regarding snitches ? What snitch do you guys use ? > > > 2014-06-05 15:06 GMT+02:00 William Oberman : > I don't think traffic will flow between "classic" ec2 and vpc directly. > There is some kind of gateway bridge instance that sits between, acting a= s > a NAT. I would think that would cause new challenges for: > -transitions > -clients > > Sorry this response isn't heavy on content! I'm curious how this thread > goes... > > Will > > On Thursday, June 5, 2014, Alain RODRIGUEZ wrote: > Hi guys, > > We are going to move from a cluster made of simple Amazon EC2 servers to = a > VPC cluster. We are using Cassandra 1.2.11 and I have some questions > regarding this switch and the Cassandra configuration inside a VPC. > > Actually I found no documentation on this topic, but I am quite sure that > some people are already using VPC. If you can point me to any documentati= on > regarding VPC / Cassandra, it would be very nice of you. We have only one > DC for now, but we need to remain multi DC compatible, since we will add = DC > very soon. > > Else, I would like to know if I should keep using EC2MultiRegionSnitch or > change the snitch to anything else. > > What about broadcast/listen ip, seeds...? > > We currently use public ip as for broadcast address and for seeds. We use > private ones for listen address. Machines inside the VPC will only have > private IP AFAIK. Should I keep using a broadcast address ? > > Is there any other incidence when switching to a VPC ? > > Sorry if the topic was already discussed, I was unable to find any useful > information... > > > -- > Will Oberman > Civic Science, Inc. > 6101 Penn Avenue, Fifth Floor > Pittsburgh, PA 15206 > (M) 412-480-7835 > (E) oberman@civicscience.com > > > > > > > > > > > > > > -- > Jon Haddad > http://www.rustyrazorblade.com > skype: rustyrazorblade > > > --001a113a2a2e057c7904fb90a788 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Tinc's developers acknowledge that there are some= fairly serious unfixed security issues in their protocol: http://www.tinc-vpn.org/security/. As suc= h, I do not consider tinc to be a good choice for production systems.

Either IPSec or OpenVPN are reasonable for connecting V= PCs in different regions, and Amazon has published guides for both methods[= 1][2]. We use IPSec because we have a lot of experience with it, but I'= m hesitant to recommend it because it is easy to configure in an insecure m= anner.



On Tue,= Jun 10, 2014 at 6:29 PM, Ben Bromhead <ben@instaclustr.com> wrote:
Have a l= ook at=C2=A0http://w= ww.tinc-vpn.org/, mesh based and handles multiple gateways for the same= network in a graceful manner (so you can run two gateways per region for H= A).

Also supports NAT traversal if you need to do public-private= clusters.=C2=A0

We are currently evaluating it fo= r our managed Cassandra in a VPC solution, but we haven=E2=80=99t ever used= it in a production environment or with a heavy load, so caveat emptor.=C2= =A0

As for the snitch=E2=80=A6 the GPFS is definitely the m= ost flexible.=C2=A0

Ben Bromhead
Instaclustr |=C2=A0<= a href=3D"https://www.instaclustr.com/" target=3D"_blank">www.instaclustr.c= om=C2=A0|=C2=A0@instaclustr=C2=A0| +61 415 936 359

On 10 Jun 2014, at 1:42 am, Ackerman, Mitchell <Mitchell.Ackerman@pgi.= com> wrote:

Peter,
=C2=A0
I too am working = on setting up a multi-region VPC Cassandra cluster.=C2=A0 Each region is co= nnected to each other via an OpenVPN tunnel, so we can use internal IP addr= esses for both the seeds and broadcast address.=C2=A0 =C2=A0This allows us = to use the EC2Snitch (my interpretation of the caveat that this snitch won= =E2=80=99t work in a multi-region environment is that it won=E2=80=99t work= if you can=E2=80=99t use internal IP addresses, which we can via the VPN t= unnels). =C2=A0All the C* nodes find each other, and nodetool (or OpsCenter= ) shows that we have established a multi-datacenter cluster.=C2=A0
=C2=A0
Thus far, I=E2=80=99m not happy with the performance of the cluster= in such a configuration, but I don=E2=80=99t think that it is related to t= his configuration, though it could be.
=C2=A0
Mitchell
= =C2=A0
From:=C2=A0Peter Sanford [mailto:psanford@retailnext.net]= =C2=A0
Sent:=C2=A0Monday, June 09, 2014 7:19 AM
To:<= span>=C2=A0user@cassandra.apache.org
Subject:=C2=A0Re= : VPC AWS
=C2=A0
=C2=A0
-Peter

=C2=A0

On Mon, Jun 9, 2014 at 5:48 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:
Hi guys, there is a lot of answer, it looks like this subject is interestin= g a lot of people, so I will end up letting you know how it went for us.=
=C2=A0
For now, we are= still doing some tests.
=C2=A0
Yet I would lik= e to know how we are supposed to configure Cassandra in this environment :<= u>
=C2=A0
- VPC=C2=A0
- Multiple= datacenters (should be VPCs, one per region, linked through VPN ?)<= u>
- Cassandra 1.2
=
=C2=A0
We are currentl= y running under EC2MultiRegionSnitch, but with no VPC. Our VPC will have no= public interface, so I am not sure how to configure broadcast address or s= eeds that are supposed to be the public IP of the node.
=C2=A0
I could use EC2Snitch, but will cross region work properly ?<= /div>
=C2=A0
Should I use an other snitch ?
=C2=A0
Is someone using a similar configuration ?
=C2=A0
Thanks for information already given guys, we will achieve this ;-).=

= =C2=A0

2014-06-07 0:05 GMT+02:00 Jonathan Haddad <<= a href=3D"mailto:jon@jonhaddad.com" style=3D"color:purple;text-decoration:u= nderline" target=3D"_blank">jon@jonhaddad.com>:
=C2=A0
This may not hel= p you with the migration, but it may with maintenance & management. =C2= =A0I just put up a blog post on managing VPC security groups with a tool I = open sourced at my previous company. =C2=A0If you're going to have diff= erent VPCs (staging / prod), it might help with managing security groups.
=C2=A0
=C2=A0
Semi shameless plug... but relevant.

=C2=A0

On Thu, Jun 5, 2014 at 12:01 PM, Aiman Parvaiz <aiman@shift.com> wrote:
Cool, thanks again for this.

=C2=A0

On Thu, Jun 5, 2014 at 11:51 AM, Michael Theroux <mtheroux2@yahoo.com> wrote:
You can have a ring spread across EC2 and the public subnet of a VPC. =C2= =A0That is how we did our migration. =C2=A0In our case, we simply replaced = the existing EC2 node with a new instance in the public VPC, restored from = a backup taken right before the switch.
= =C2=A0
-Mike
=C2=A0
From:=C2=A0Aiman Parvaiz <aiman@shift.com>
To:=C2=A0Michael Theroux <mtheroux2@yahoo.com>=C2=A0
Cc:=C2= =A0"user@cassandra.apache.= org" <user@cassandra.apach= e.org>=C2=A0
Sent:=C2=A0Thursday, June 5, 2014 2:39 PM
Subject= :=C2=A0Re: VPC AWS
=C2=A0
Thanks for this = info Michael. As far as restoring node in public VPC is concerned I was thi= nking ( and I might be wrong here) if we can have a ring spread across EC2 = and public subnet of a VPC, this way I can simply decommission nodes in Ec2= as I gradually introduce new nodes in public subnet of VPC and I will end = up with a ring in public subnet and then migrate them from public to privat= e in a similar way may be.
=C2=A0
If anyone has any experience/ suggestions with this please share, would rea= lly appreciate it.
= =C2=A0
Aiman

=C2=A0

On Thu, Jun 5, 2014 at = 10:37 AM, Michael Theroux <mtheroux2@yah= oo.com> wrote:
The implementation of moving from EC2 to a= VPC was a bit of a juggling act. =C2=A0Our motivation was two fold:=
= =C2=A0
1) We were running out of static IP= addresses, and it was becoming increasingly difficult in EC2 to design aro= und limiting the number of static IP addresses to the number of public IP a= ddresses EC2 allowed
2= ) VPC affords us an additional level of security that was desirable.=
= =C2=A0
However, we needed to consider the = following limitations:
=C2=A0
1) By default, yo= u have a limited number of available public IPs for both EC2 and VPC. =C2= =A0
2= ) AWS security groups need to be configured to allow traffic for Cassandra = to/from instances in EC2 and the VPC.
= =C2=A0
You are correct at the high level t= hat the migration goes from EC2->Public VPC (VPC with an Internet Gatewa= y)->Private VPC (VPC with a NAT). =C2=A0The first phase was moving insta= nces to the public VPC, setting broadcast and seeds to the public IPs we ha= d available. =C2=A0Basically:
= =C2=A0
1) Take down a node, taking a snaps= hot for a backup
2) Restore the node on the public V= PC, assigning it to the correct security group, manually setting the seeds = to other available nodes
3) Verify the cluster can communica= te
4) Repeat
= =C2=A0
Realize the NAT instance on the pri= vate subnet will also require a public IP. =C2=A0What got really interestin= g is that near the end of the process we ran out of available IPs, requirin= g us to switch the final node that was on EC2 directly to the private VPC (= and taking down two nodes at once, which our setup allowed given we had 6 n= odes with an RF of 3). =C2=A0
= =C2=A0
What we did, and highly suggest for= the switch, is to write down every step that has to happen on every node d= uring the switch. =C2=A0In our case, many of the moved nodes required sligh= tly different configurations for items like the seeds.=
= =C2=A0
Its been a couple of years, so my m= emory on this maybe a little fuzzy :)
=
=C2=A0
-Mike
=C2=A0
From:=C2=A0Aiman Parvaiz <aiman@shift.com>
To:=C2=A0user@cass= andra.apache.org; Michael Theroux <mt= heroux2@yahoo.com>=C2=A0
Sent:=C2=A0Thursday, June 5, 2014 12:55 PM
Subjec= t:=C2=A0Re: VPC AWS
<= div style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times = New Roman',serif"> =C2=A0
Michael,=C2=A0
Thanks for the response, I am about to head in to something very similar if= not exactly same. I envision things happening on the same lines as you men= tioned.=C2=A0
I would be grateful if you could please throw some more light on how you we= nt about switching cassandra nodes from public subnet to private with out a= ny downtime.
I have not started on this project yet, still in my research phase. I plan = to have a ec2+public VPC cluster and then decomission ec2 nodes to have eve= rything in public subnet, next would be to move it to private subnet.
=C2=A0
Thanks

=C2=A0

On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux <mtheroux2@yahoo.com> wrote:
We personally use the EC2Snitch, however, we don't have the multi-regio= n requirements you do,
=C2=A0
-Mike
=C2=A0
From:=C2=A0Alain RODRIGUEZ <arodrime@gmail.com>
To:=C2=A0user@cass= andra.apache.org
Sent:=C2=A0Thursday, June 5, 2014 9:14 AM
Subject= :=C2=A0Re: VPC AWS
=C2=A0
I think you can = define VPC subnet to be public (to have public + private IPs) or private on= ly.
=C2=A0
Any insight regarding snitches ? What snitch do you guys use ?

=C2=A0<= u>

2014-06-05 15:06 GMT+02:00 William Oberman= <oberman@civicscience.com>:<= u>
I don't think traffic will flow between &qu= ot;classic" ec2 and vpc directly. There is some kind of gateway bridge= instance that sits between, acting as a NAT. =C2=A0 I would think that wou= ld cause new=C2=A0challenges for:
-transitions=C2=A0
-clients
=C2=A0=
Sorry this response isn't heavy on content! =C2=A0I'm curious how t= his thread goes...
= =C2=A0
Will
<= div style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times = New Roman',serif"> =C2=A0
<= /div>
=C2=A0
Sorry if the topic was already discussed, I was unable to find any useful i= nformation...

=C2=A0

=C2=A0
=C2=A0

<= /u>=C2=A0

=C2=A0

<= /u>=C2=A0

=C2=A0

<= /u>=C2=A0


=C2=A0
--=C2=A0
Jon Haddad<= br>http://www.rustyrazorblade.com
skype: r= ustyrazorblade


--001a113a2a2e057c7904fb90a788--