aurora-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wfar...@apache.org
Subject aurora git commit: Add a guide for installing Aurora.
Date Mon, 21 Dec 2015 18:56:16 GMT
Repository: aurora
Updated Branches:
  refs/heads/master d3dae19d5 -> cf225b793


Add a guide for installing Aurora.

Bugs closed: AURORA-1483

Reviewed at https://reviews.apache.org/r/41564/


Project: http://git-wip-us.apache.org/repos/asf/aurora/repo
Commit: http://git-wip-us.apache.org/repos/asf/aurora/commit/cf225b79
Tree: http://git-wip-us.apache.org/repos/asf/aurora/tree/cf225b79
Diff: http://git-wip-us.apache.org/repos/asf/aurora/diff/cf225b79

Branch: refs/heads/master
Commit: cf225b793d5fc414e9a06f97e57ffc5301df7e58
Parents: d3dae19
Author: Bill Farner <wfarner@apache.org>
Authored: Mon Dec 21 10:56:05 2015 -0800
Committer: Bill Farner <wfarner@apache.org>
Committed: Mon Dec 21 10:56:05 2015 -0800

----------------------------------------------------------------------
 README.md          |   2 +-
 docs/installing.md | 284 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 285 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/aurora/blob/cf225b79/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index e88e1be..143ee06 100644
--- a/README.md
+++ b/README.md
@@ -72,7 +72,7 @@ Are you using Aurora too?  Let us know, or submit a patch to join the list!
 ## Getting Started
 * [Try the tutorial](docs/tutorial.md)
 * [Running a Local Cluster with Vagrant](docs/vagrant.md)
-* [Deploying Aurora](docs/deploying-aurora-scheduler.md)
+* [Installing Aurora](docs/installing.md)
 * [Developing Aurora](docs/developing-aurora-scheduler.md)
 
 ## Getting Help

http://git-wip-us.apache.org/repos/asf/aurora/blob/cf225b79/docs/installing.md
----------------------------------------------------------------------
diff --git a/docs/installing.md b/docs/installing.md
new file mode 100644
index 0000000..8d73725
--- /dev/null
+++ b/docs/installing.md
@@ -0,0 +1,284 @@
+# Installing Aurora
+
+- [Components](#components)
+  - [Machine profiles](#machine-profiles)
+    - [Coordinator](#coordinator)
+    - [Worker](#worker)
+    - [Client](#client)
+  - [Getting Aurora](#getting-aurora)
+    - [Building your own binary packages](#building-your-own-binary-packages)
+    - [RPMs](#rpms)
+  - [Installing the scheduler](#installing-the-scheduler)
+    - [Ubuntu 14.04](#ubuntu-1404)
+    - [CentOS 7](#centos-7)
+    - [Finalizing](#finalizing)
+    - [Configuration](#configuration)
+  - [Installing worker components](#installing-worker-components)
+    - [Ubuntu 14.04](#ubuntu-1404-1)
+    - [CentOS 7](#centos-7-1)
+    - [Configuration](#configuration-1)
+  - [Installing the client](#installing-the-client)
+    - [Ubuntu 14.04](#ubuntu-1404-2)
+    - [CentOS 7](#centos-7-2)
+    - [Configuration](#configuration-2)
+  - [See also](#see-also)
+  - [Installing Mesos](#installing-mesos)
+    - [Mesos on Ubuntu 14.04](#mesos-on-ubuntu-1404)
+    - [Mesos on CentOS 7](#mesos-on-centos-7)
+
+<!-- TODO(wfarner): Remove notes about "older RPM versions" once >=0.10.0 RPMs are
released. -->
+
+## Components
+Before installing Aurora, it's important to have an understanding of the components that
make up
+a functioning Aurora cluster.
+
+* **Aurora scheduler**
+
+  The scheduler will be your primary interface to the work you run in your cluster.  You
will
+  instruct it to run jobs, and it will manage them in Mesos for you.  You will also frequently
use
+  the scheduler's web interface as a heads-up display for what's running in your cluster.
+
+* **Aurora client**
+
+  The client (`aurora` command) is a command line tool that exposes primitives that you can
use to
+  interact with the scheduler.
+
+  Aurora also provides an admin client (`aurora_admin` command) that contains commands built
for
+  cluster administrators.  You can use this tool to do things like manage user quotas and
manage
+  graceful maintenance on machines in cluster.
+
+* **Aurora executor**
+
+  The executor (a.k.a. Thermos executor) is responsible for carrying out the workloads described
in
+  the Aurora DSL (`.aurora` files).  The executor is what actually executes user processes.
 It will
+  also perform health checking of tasks and register tasks in ZooKeeper for the purposes
of dynamic
+  service discovery.  You can find lots more detail on the executor and Thermos in the
+  [user guide](user-guide.md).
+
+* **Aurora observer**
+
+  The observer provides browser-based access to the status of individual tasks executing
on worker
+  machines.  It gives insight into the processes executing, and facilitates browsing of task
sandbox
+  directories.
+
+* **ZooKeeper**
+
+  [ZooKeeper](http://zookeeper.apache.org) is a distributed consensus system.  In an Aurora
cluster
+  it is used for reliable election of the leading Aurora scheduler and Mesos master.
+
+* **Mesos master**
+
+  The master is responsible for tracking worker machines and performing accounting of their
+  resources.  The scheduler interfaces with the master to control the cluster.
+
+* **Mesos agent**
+
+  The agent receives work assigned by the scheduler and executes them.  It interfaces with
Linux
+  isolation systems like cgroups, namespaces and Docker to manage the resource consumption
of tasks.
+  When a user task is launched, the agent will launch the executor (in the context of a Linux
cgroup
+  or Docker container depending upon the environment), which will in turn fork user processes.
+
+## Machine profiles
+Given that many of these components communicate over the network, there are numerous ways
you could
+assemble them to create an Aurora cluster.  The simplest way is to think in terms of three
machine
+profiles:
+
+### Coordinator
+**Components**: ZooKeeper, Aurora scheduler, Mesos master
+
+A small number of machines (typically 3 or 5) responsible for cluster orchestration.  In
most cases
+it is fine to co-locate these components in anything but very large clusters (> 1000 machines).
+Beyond that point, operators will likely want to manage these services on separate machines.
+
+In practice, 5 coordinators have been shown to reliably manage clusters with tens of thousands
of
+machines.
+
+
+### Worker
+**Components**: Aurora executor, Aurora observer, Mesos agent
+
+The bulk of the cluster, where services will actually run.
+
+### Client
+**Components**: Aurora client, Aurora admin client
+
+Any machines that users submit jobs from.
+
+## Getting Aurora
+Source and binary distributions can be found on our
+[downloads](https://aurora.apache.org/downloads/) page.  Installing from binary packages
is
+recommended for most.
+
+### Building your own binary packages
+Our package build toolchain makes it easy to build your own packages if you would like. 
See the
+[instructions](https://github.com/apache/aurora-packaging) to learn how.
+
+### RPMs
+We currently have work in progress to provide official RPMs.  As of this writing, the suggested
way
+to get RPMs is to [build them](#building-your-own-binary-packages).
+
+We do have unofficial experimental RPMs available for testing purposes.
+
+**Use these RPMs at your own risk, they are not officially released under the ASF guidelines.**
+
+    echo '[apache-aurora-wfarner]
+    name=Apache Aurora distribution maintained by wfarner
+    baseurl=http://people.apache.org/~wfarner/aurora/distributions/0.9.0/rpm/centos-7/x86_64/
+    gpgcheck = 0' | sudo tee /etc/yum.repos.d/apache-aurora-wfarner.repo > /dev/null
+
+
+## Installing the scheduler
+### Ubuntu 14.04
+
+1. Install Mesos
+
+   Skip down to [install mesos](#mesos-on-ubuntu-1404), then run:
+
+        sudo start mesos-master
+
+2. Install ZooKeeper
+
+        sudo apt-get install -y zookeeperd
+
+3. Install the Aurora scheduler
+
+        wget -c https://apache.bintray.com/aurora/aurora-scheduler_0.10.0-1_amd64.deb
+        sudo dpkg -i aurora-scheduler_0.10.0-1_amd64.deb
+
+### CentOS 7
+
+1. Install Mesos
+
+   Skip down to [install mesos](#mesos-on-centos-7), then run:
+
+        sudo systemctl start mesos-master
+
+2. Install ZooKeeper
+
+        sudo rpm -Uvh https://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm
+        sudo yum install -y java-1.8.0-openjdk-headless zookeeper-server
+
+        sudo service zookeeper-server init
+        sudo systemctl start zookeeper-server
+
+3. Install the Aurora scheduler  
+   If you haven't already, read the section on [how to get Aurora RPMs](#rpms).
+
+        # Note: for older Aurora RPM versions, this may be called 'aurora'.
+        sudo yum install -y aurora-scheduler
+
+Note: if you are using the unreleased 0.9.0 RPM, you will need to edit `/etc/sysconfig/aurora`:
 
+Change  
+`-mesos_master_address='zk://127.0.0.1:2181/mesos/master'`  
+To  
+`-mesos_master_address='zk://127.0.0.1:2181/mesos'`  
+And  
+`-native_log_file_path='/var/lib/aurora/db'`  
+To  
+`-native_log_file_path='/var/lib/aurora/scheduler/db'`
+
+### Finalizing
+By default, the scheduler will start in an uninitialized mode.  This is because external
+coordination is necessary to be certain operator error does not result in a quorum of schedulers
+starting up and believing their databases are empty when in fact they should be re-joining
a
+cluster.
+
+Because of this, a fresh install of the scheduler will need intervention to start up.  First,
+stop the scheduler service.  
+Ubuntu: `sudo stop aurora-scheduler`  
+CentOS: `sudo systemctl stop aurora`
+
+Now initialize the database:
+
+    sudo -u aurora mkdir -p /var/lib/aurora/scheduler/db
+    sudo -u aurora mesos-log initialize --path=/var/lib/aurora/scheduler/db
+
+Now you can start the scheduler back up.  
+Ubuntu: `sudo start aurora-scheduler`  
+CentOS: `sudo systemctl start aurora`
+
+### Configuration
+For more detail on this topic, see the dedicated page on
+[deploying the scheduler](deploying-aurora-scheduler.md)
+
+
+## Installing worker components
+### Ubuntu 14.04
+
+1. Install Mesos
+
+   Skip down to [install mesos](#mesos-on-ubuntu-1404), then run:
+
+        sudo start mesos-slave
+
+2. Install Aurora executor and observer
+
+        wget -c https://apache.bintray.com/aurora/aurora-executor_0.10.0-1_amd64.deb
+        sudo dpkg -i aurora-executor_0.10.0-1_amd64.deb
+
+### CentOS 7
+
+1. Install Mesos
+
+   Skip down to [install mesos](#mesos-on-centos-7), then run:
+
+        sudo systemctl start mesos-slave
+
+2. Install Aurora executor and observer  
+   If you haven't already, read the section on [how to get Aurora RPMs](#rpms).
+
+        # Note: for older Aurora RPM versions, this may be called 'aurora-thermos'.
+        sudo yum install -y aurora-executor
+
+### Configuration
+The executor and observer typically do not require much configuration.  Command line arguments
can
+be passed to the executor using a command line argument on the scheduler.
+
+## Installing the client
+### Ubuntu 14.04
+
+        sudo apt-get install -y python2.7 wget
+
+        wget https://apache.bintray.com/aurora/aurora-tools_0.10.0-1_amd64.deb
+        sudo dpkg -i aurora-tools_0.10.0-1_amd64.deb
+
+### CentOS 7
+If you haven't already, read the section on [how to get Aurora RPMs](#rpms).
+
+        # Note: for older Aurora RPM versions, this may be called 'aurora-client'.
+        sudo yum install -y aurora-tools
+
+### Configuration
+Client configuration lives in a json file that describes the clusters available and how to
reach
+them.  By default this file is at `/etc/aurora/clusters.json`.
+
+Jobs may be submitted to the scheduler using the client, and are described with
+[job configurations](configuration-reference.md) expressed in `.aurora` files.  Typically
you will
+maintain a single job configuration file to describe one or more deployment environments
(e.g.
+dev, test, prod) for a production job.
+
+## See also
+We have other docs that you will find useful once you have your cluster up and running:
+* [Monitor](monitoring.md) your cluster
+* Enable scheduler [security](security.md)
+* View job SLA [statistics](sla.md)
+* Understand the internals of the scheduler's [storage](storage.md)
+
+## Installing Mesos
+Mesos uses a single package for the Mesos master and slave.  As a result, the package dependencies
+are identical for both.
+
+### Mesos on Ubuntu 14.04
+
+    sudo add-apt-repository ppa:openjdk-r/ppa -y
+    sudo apt-get update
+
+    sudo apt-get install -y software-properties-common wget libsvn1 libcurl3 openjdk-8-jre-headless
+
+    wget -c http://downloads.mesosphere.io/master/ubuntu/14.04/mesos_0.23.0-1.0.ubuntu1404_amd64.deb
+    sudo dpkg -i mesos_0.23.0-1.0.ubuntu1404_amd64.deb
+
+### Mesos on CentOS 7
+
+    sudo rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
+    sudo yum install -y mesos-0.22.0


Mime
View raw message