Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2190A1151A for ; Mon, 9 Jun 2014 12:48:54 +0000 (UTC) Received: (qmail 84208 invoked by uid 500); 9 Jun 2014 12:48:51 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 84173 invoked by uid 500); 9 Jun 2014 12:48:51 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 84165 invoked by uid 99); 9 Jun 2014 12:48:51 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Jun 2014 12:48:51 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of arodrime@gmail.com designates 209.85.215.51 as permitted sender) Received: from [209.85.215.51] (HELO mail-la0-f51.google.com) (209.85.215.51) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Jun 2014 12:48:47 +0000 Received: by mail-la0-f51.google.com with SMTP id gf5so2967310lab.24 for ; Mon, 09 Jun 2014 05:48:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=vzLlq3rrRk5CgyglX15EJhPWi4TOGXNdN42FaxH8UHA=; b=EZbAzYua+tMMBwrIxwy4th6oDxMqslPk7UHo2BdIt+DshePBViib7UD9nzC5gXbroW rtO6QNCa5/CWWE0oj2KL2TUOIDQbxAUVWrXesClohwHVzC0bAL+Q7UPX7HNYlRFfiZN0 VGrlzen2GKiM16QtnDUD9WIEe9PWiKb8N+JtVJU99Q4z30WN2dtPU+74TxhoMKenYE7H LJVYNaRm0ldDhM4SxLl91VC7h2+9HX4Uzj6Cm6Q9oE9Y/BxgNPXWGkoxiFpsPpq/2C5W H6d1N/T/mIjniDBLWnthemKzxRvof/6azM+gJ8gYZNGDT2Qw0qZ58UYvfPnHkyaig0Ih F00w== X-Received: by 10.112.164.148 with SMTP id yq20mr16945812lbb.22.1402318106187; Mon, 09 Jun 2014 05:48:26 -0700 (PDT) MIME-Version: 1.0 Received: by 10.112.215.106 with HTTP; Mon, 9 Jun 2014 05:48:06 -0700 (PDT) In-Reply-To: References: <1401981267.29077.YahooMailNeo@web121903.mail.ne1.yahoo.com> <1401989875.13156.YahooMailNeo@web121906.mail.ne1.yahoo.com> <1401994290.53645.YahooMailNeo@web121905.mail.ne1.yahoo.com> From: Alain RODRIGUEZ Date: Mon, 9 Jun 2014 14:48:06 +0200 Message-ID: Subject: Re: VPC AWS To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a1133e32c94dad104fb66a039 X-Virus-Checked: Checked by ClamAV on apache.org --001a1133e32c94dad104fb66a039 Content-Type: text/plain; charset=ISO-8859-1 Hi guys, there is a lot of answer, it looks like this subject is interesting a lot of people, so I will end up letting you know how it went for us. For now, we are still doing some tests. Yet I would like to know how we are supposed to configure Cassandra in this environment : - VPC - Multiple datacenters (should be VPCs, one per region, linked through VPN ?) - Cassandra 1.2 We are currently running under EC2MultiRegionSnitch, but with no VPC. Our VPC will have no public interface, so I am not sure how to configure broadcast address or seeds that are supposed to be the public IP of the node. I could use EC2Snitch, but will cross region work properly ? Should I use an other snitch ? Is someone using a similar configuration ? Thanks for information already given guys, we will achieve this ;-). 2014-06-07 0:05 GMT+02:00 Jonathan Haddad : > This may not help you with the migration, but it may with maintenance & > management. I just put up a blog post on managing VPC security groups with > a tool I open sourced at my previous company. If you're going to have > different VPCs (staging / prod), it might help with managing security > groups. > > http://rustyrazorblade.com/2014/06/an-introduction-to-roadhouse/ > > Semi shameless plug... but relevant. > > > On Thu, Jun 5, 2014 at 12:01 PM, Aiman Parvaiz wrote: > >> Cool, thanks again for this. >> >> >> On Thu, Jun 5, 2014 at 11:51 AM, Michael Theroux >> wrote: >> >>> You can have a ring spread across EC2 and the public subnet of a VPC. >>> That is how we did our migration. In our case, we simply replaced the >>> existing EC2 node with a new instance in the public VPC, restored from a >>> backup taken right before the switch. >>> >>> -Mike >>> >>> ------------------------------ >>> *From:* Aiman Parvaiz >>> *To:* Michael Theroux >>> *Cc:* "user@cassandra.apache.org" >>> *Sent:* Thursday, June 5, 2014 2:39 PM >>> *Subject:* Re: VPC AWS >>> >>> Thanks for this info Michael. As far as restoring node in public VPC is >>> concerned I was thinking ( and I might be wrong here) if we can have a ring >>> spread across EC2 and public subnet of a VPC, this way I can simply >>> decommission nodes in Ec2 as I gradually introduce new nodes in public >>> subnet of VPC and I will end up with a ring in public subnet and then >>> migrate them from public to private in a similar way may be. >>> >>> If anyone has any experience/ suggestions with this please share, would >>> really appreciate it. >>> >>> Aiman >>> >>> >>> On Thu, Jun 5, 2014 at 10:37 AM, Michael Theroux >>> wrote: >>> >>> The implementation of moving from EC2 to a VPC was a bit of a juggling >>> act. Our motivation was two fold: >>> >>> 1) We were running out of static IP addresses, and it was becoming >>> increasingly difficult in EC2 to design around limiting the number of >>> static IP addresses to the number of public IP addresses EC2 allowed >>> 2) VPC affords us an additional level of security that was desirable. >>> >>> However, we needed to consider the following limitations: >>> >>> 1) By default, you have a limited number of available public IPs for >>> both EC2 and VPC. >>> 2) AWS security groups need to be configured to allow traffic for >>> Cassandra to/from instances in EC2 and the VPC. >>> >>> You are correct at the high level that the migration goes from >>> EC2->Public VPC (VPC with an Internet Gateway)->Private VPC (VPC with a >>> NAT). The first phase was moving instances to the public VPC, setting >>> broadcast and seeds to the public IPs we had available. Basically: >>> >>> 1) Take down a node, taking a snapshot for a backup >>> 2) Restore the node on the public VPC, assigning it to the correct >>> security group, manually setting the seeds to other available nodes >>> 3) Verify the cluster can communicate >>> 4) Repeat >>> >>> Realize the NAT instance on the private subnet will also require a >>> public IP. What got really interesting is that near the end of the >>> process we ran out of available IPs, requiring us to switch the final node >>> that was on EC2 directly to the private VPC (and taking down two nodes at >>> once, which our setup allowed given we had 6 nodes with an RF of 3). >>> >>> What we did, and highly suggest for the switch, is to write down every >>> step that has to happen on every node during the switch. In our case, many >>> of the moved nodes required slightly different configurations for items >>> like the seeds. >>> >>> Its been a couple of years, so my memory on this maybe a little fuzzy :) >>> >>> -Mike >>> >>> ------------------------------ >>> *From:* Aiman Parvaiz >>> *To:* user@cassandra.apache.org; Michael Theroux >>> *Sent:* Thursday, June 5, 2014 12:55 PM >>> *Subject:* Re: VPC AWS >>> >>> Michael, >>> Thanks for the response, I am about to head in to something very similar >>> if not exactly same. I envision things happening on the same lines as you >>> mentioned. >>> I would be grateful if you could please throw some more light on how you >>> went about switching cassandra nodes from public subnet to private with out >>> any downtime. >>> I have not started on this project yet, still in my research phase. I >>> plan to have a ec2+public VPC cluster and then decomission ec2 nodes to >>> have everything in public subnet, next would be to move it to private >>> subnet. >>> >>> Thanks >>> >>> >>> On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux >>> wrote: >>> >>> We personally use the EC2Snitch, however, we don't have the multi-region >>> requirements you do, >>> >>> -Mike >>> >>> ------------------------------ >>> *From:* Alain RODRIGUEZ >>> *To:* user@cassandra.apache.org >>> *Sent:* Thursday, June 5, 2014 9:14 AM >>> *Subject:* Re: VPC AWS >>> >>> I think you can define VPC subnet to be public (to have public + private >>> IPs) or private only. >>> >>> Any insight regarding snitches ? What snitch do you guys use ? >>> >>> >>> 2014-06-05 15:06 GMT+02:00 William Oberman : >>> >>> I don't think traffic will flow between "classic" ec2 and vpc directly. >>> There is some kind of gateway bridge instance that sits between, acting as >>> a NAT. I would think that would cause new challenges for: >>> -transitions >>> -clients >>> >>> Sorry this response isn't heavy on content! I'm curious how this thread >>> goes... >>> >>> Will >>> >>> On Thursday, June 5, 2014, Alain RODRIGUEZ wrote: >>> >>> Hi guys, >>> >>> We are going to move from a cluster made of simple Amazon EC2 servers to >>> a VPC cluster. We are using Cassandra 1.2.11 and I have some questions >>> regarding this switch and the Cassandra configuration inside a VPC. >>> >>> Actually I found no documentation on this topic, but I am quite sure >>> that some people are already using VPC. If you can point me to any >>> documentation regarding VPC / Cassandra, it would be very nice of you. We >>> have only one DC for now, but we need to remain multi DC compatible, since >>> we will add DC very soon. >>> >>> Else, I would like to know if I should keep using EC2MultiRegionSnitch >>> or change the snitch to anything else. >>> >>> What about broadcast/listen ip, seeds...? >>> >>> We currently use public ip as for broadcast address and for seeds. We >>> use private ones for listen address. Machines inside the VPC will only have >>> private IP AFAIK. Should I keep using a broadcast address ? >>> >>> Is there any other incidence when switching to a VPC ? >>> >>> Sorry if the topic was already discussed, I was unable to find any >>> useful information... >>> >>> >>> >>> -- >>> Will Oberman >>> Civic Science, Inc. >>> 6101 Penn Avenue, Fifth Floor >>> Pittsburgh, PA 15206 >>> (M) 412-480-7835 >>> (E) oberman@civicscience.com >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> > > > -- > Jon Haddad > http://www.rustyrazorblade.com > skype: rustyrazorblade > --001a1133e32c94dad104fb66a039 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi guys, there is a lot of answer, it looks like this subj= ect is interesting a lot of people, so I will end up letting you know how i= t went for us.

For now, we are still doing some tests.

Yet I would like to know how we are supposed to configu= re Cassandra in this environment :

- VPC=A0
<= div>- Multiple datacenters (should be VPCs, one per region, linked through = VPN ?)
- Cassandra 1.2

We are currently running unde= r EC2MultiRegionSnitch, but with no VPC. Our VPC will have no public interf= ace, so I am not sure how to configure broadcast address or seeds that are = supposed to be the public IP of the node.

I could use EC2Snitch, but will cross region work prope= rly ?

Should I use an other snitch ?

Is someone using a similar configuration ?

Thanks for information already given guys, we will achieve this = ;-).


2014-06-07 0:05 GMT+02:00 Jonathan Haddad <jon@jonhaddad.com>:
This may not help you = with the migration, but it may with maintenance & management. =A0I just= put up a blog post on managing VPC security groups with a tool I open sour= ced at my previous company. =A0If you're going to have different VPCs (= staging / prod), it might help with managing security groups.


Semi s= hameless plug... but relevant.


On Thu, Jun 5, 2014 at 12:01 PM, Aiman Parvaiz <aiman@shi= ft.com> wrote:
Cool, thanks again for this= .


On Thu, Jun 5, 2014 at 11:51 AM, Michael Theroux <mtheroux2@yahoo.com> wrote:
You can have a ring spr= ead across EC2 and the public subnet of a VPC. =A0That is how we did our mi= gration. =A0In our case, we simply replaced the existing EC2 node with a ne= w instance in the public VPC, restored from a backup taken right before the= switch.

-Mike

From: Aiman Parvaiz <aiman@shift.com>
To: Michael Theroux <mtheroux2@yahoo.com>
Cc: "user@cassandra.apache.org&= quot; <us= er@cassandra.apache.org>
Sent: Thursday, June 5, 201= 4 2:39 PM
Subject: Re: V= PC AWS

Thanks for this info Michael. As far as restoring node in public VPC is con= cerned I was thinking ( and I might be wrong here) if we can have a ring sp= read across EC2 and public subnet of a VPC, this way I can simply decommiss= ion nodes in Ec2 as I gradually introduce new nodes in public subnet of VPC= and I will end up with a ring in public subnet and then migrate them from public to private in a similar way may b= e.

If anyone has any experience/ suggestions wit= h this please share, would really appreciate it.

Aiman


On Thu, Jun 5, 2014 at 10:37 AM, Michael Theroux <= ;mtheroux2@yahoo.com> wrote:
The im= plementation of moving from EC2 to a VPC was a bit of a juggling act. =A0Ou= r motivation was two fold:

=
1) We were running out of static IP addresses, and it was becoming in= creasingly difficult in EC2 to design around limiting the number of static = IP addresses to the number of public IP addresses EC2 allowed
2) VPC affords us an additional level of security that was desirable.
However, we needed to consider the following limitations:
1) By default, you have a limited number of available public IPs for = both EC2 and VPC. =A0
2) AWS security groups need to be configured to allow traffic for Cassandra to/from instances in EC2 and the= VPC.

You are correct at the high level that the migration goes from EC2-&g= t;Public VPC (VPC with an Internet Gateway)->Private VPC (VPC with a NAT= ). =A0The first phase was moving instances to the public VPC, setting broad= cast and seeds to the public IPs we had available. =A0Basically:

=
1) Take down a node, taking a = snapshot for a backup
2) Restore the node on the public VPC, assigning it to the correct se= curity group, manually setting the seeds to other available nodes
3) Verify the cluster can communicate
4) Repeat

= Realize the NAT instance on the privat= e subnet will also require a public IP. =A0What got really interesti= ng is that near the end of the process we ran out of available IPs, requiri= ng us to switch the final node that was on EC2 directly to the private VPC = (and taking down two nodes at once, which our setup allowed given we had 6 = nodes with an RF of 3). =A0

=
What we did, and highly suggest for the switch, is to write down ever= y step that has to happen on every node during the switch. =A0In our case, = many of the moved nodes required slightly different configurations for item= s like the seeds.

=
Its been a couple of years, so my memory on this maybe a little fuzzy :)

-Mike

From: Aiman Parvaiz <aiman@shift.com>
To: use= r@cassandra.apache.org; Michael Theroux <mtheroux2@y= ahoo.com>
Sent: Thursday, June 5, 201= 4 12:55 PM
Subject:<= /span> Re: VPC AWS

Michael,=A0
Thanks for the response, I am about to head in to something= very similar if not exactly same. I envision things happening on the same = lines as you mentioned.=A0
I would be grateful if you could pleas= e throw some more light on how you went about switching cassandra nodes fro= m public subnet to private with out any downtime.
I have not started on this project yet, still in my research phase. I = plan to have a ec2+public VPC cluster and then decomission ec2 nodes to hav= e everything in public subnet, next would be to move it to private subnet.<= /div>

Thanks


On Thu, Jun 5, 2014 at 8:14 AM, Michael Ther= oux <mtheroux2@yahoo.com> = wrote:
We personally use the EC2Snitch, however, we = don't have the multi-region requirements you do,

=
-Mike

=
From: Alain RODRIGUEZ <<= a rel=3D"nofollow" shape=3D"rect" href=3D"mailto:arodrime@gmail.com" target= =3D"_blank">arodrime@gmail.com>
To: use= r@cassandra.apache.org
Sent: Thursday, June 5, 2014 9:14 AM
Subject: Re: VPC AWS

I think you can define VPC subnet to be public (to have public + p= rivate IPs) or private only.

Any insight regarding snitches ? What snitch = do you guys use ?


2014-06-05 15:06 GMT+02:00 = William Oberman <oberman@civicscien= ce.com>:
I don't think traffic will flow between "classic" ec2 an= d vpc directly. There is some kind of gateway bridge instance that sits bet= ween, acting as a NAT. =A0 I would think that would cause new=A0challenges = for:
-transitions=A0
-clients

=
Sorry this response isn't heavy on content! =A0I'm curious ho= w this thread goes...

Will

On Thursday, June 5, 2014, Alain RODRI= GUEZ <arodrime@gmail.com> wrote:
Hi guys,

We are= going to move from a cluster made of simple Amazon EC2 servers to a VPC cl= uster. We are using Cassandra 1.2.11 and I have some questions regarding th= is switch and the Cassandra configuration inside a VPC.

Actually I found no documentation on thi= s topic, but I am quite sure that some people are already using VPC. If you= can point me to any documentation regarding VPC / Cassandra, it would be v= ery nice of you. We have only one DC for now, but we need to remain multi D= C compatible, since we will add DC very soon.

Else, I would like to know if I should k= eep using EC2MultiRegionSnitch or change the snitch to anything else.
=

What about broadcast/listen ip, seeds...= ?

We currently use public ip as for broadcast address and for seeds. We use p= rivate ones for listen address. Machines inside the VPC will only have priv= ate IP AFAIK. Should I keep using a broadcast address ?

Is there any other incidence when switching to a VPC ?

Sorry if the topic was already discussed, I was un= able to find any useful information...


= --
Will Oberman
Civic Science, Inc.
6101 Penn Avenue, Fifth Flo= or
Pittsburgh, PA 15206
(M) 412-480-7835
(E) oberman@civicscience.com








=



<= /div>--
Jon Haddad
http://www.rusty= razorblade.com
skype: rustyrazorblade
--001a1133e32c94dad104fb66a039--