aurora-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Staffin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AURORA-1618) Deploying Aurora on AWS
Date Fri, 12 Feb 2016 22:10:19 GMT

    [ https://issues.apache.org/jira/browse/AURORA-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145400#comment-15145400
] 

Benjamin Staffin commented on AURORA-1618:
------------------------------------------

Some brain-dumping:

h5. Instance types:
It depends on your workloads of course, but in general the largest instances of whatever type
you choose will be the most cost effective, since mesos means you can binpack jobs onto them
pretty effectively.  T2.large instances are great for dev clusters that aren't cpu loaded.
M4 are a decent default choice for production clusters, or C4 if your workloads are CPU-bound.

h5. Regions/AZs:
I've been doing region==cluster, and labeling slave nodes with what Availability Zone they
fall in to allow sensible scheduling constraints.  You'll want to distribute your master nodes
across AZs if possible.  EBS isn't really necessary, but the instance types you'll probably
want to use for new deploys (M4, maybe T2 for development) are EBS-only.

h5. IPs/Hostnames:
Run everything in a VPC.  If you have ec2's auto-assigned hostnames enabled, and your instances
are assigned external IPs, they will have two hostnames: "public dns" and "private dns". 
From inside the VPC, the "public dns" hostname will always resolve to the instance's private
IP address, so you can run all the mesos/aurora components with {{--hostname=<public_dns_hostname>}}
and benefit from the hostnames all ending with .amazonaws.com, which can make things like
VPN configuration simpler.

h5. Finding zookeeper
Since you're on AWS, you should consider launching all your components in autoscaling groups
(with fixed instance counts, not dynamic), including Zookeeper.  To get a predictable hostname
for your zookeeper ensemble you can use ELB (zookeeper works behind a TCP load balancer) or
a cronjob that updates a Route53 dns entry to point at your zookeeper nodes.  Exhibitor makes
running zookeeper in autoscaling groups very convenient, and provides an http endpoing listing
the ensemble members, which you can then sync to route53 with something like https://github.com/benley/exhibitor2dns.

h5. Finding the mesos/aurora masters
Use Zookeeper where possible.  If you're using Mesos-DNS, that will give you a consistent
hostname for your mesos masters.  You could also use something like Aurproxy to set up a http
reverse proxy for your mesos and aurora master nodes.  If you run your mesos/aurora masters
in autoscaling groups too, they could also potentially go behind ELBs.

> Deploying Aurora on AWS
> -----------------------
>
>                 Key: AURORA-1618
>                 URL: https://issues.apache.org/jira/browse/AURORA-1618
>             Project: Aurora
>          Issue Type: Story
>          Components: Documentation
>            Reporter: Stephan Erb
>              Labels: documentation
>
> AWS tends to be one of the more popular deployment targets these days. Especially for
new adopters of Aurora (and Mesos), this can be quite tricky. 
> We should provide a high level guide covering the basic ground necessary for a solid
deployment.
> Potential issues/questions that should probably be covered:
> * recommended instance type?
> * mapping of components onto regions and availability zones?
> * do I need EBS for my master and scheduler nodes?
> * recommended way how to bind ip/hostnames of the different components, so that they
can reliably talk to each other
> * ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message