; Wed, 6 Jul 2016 21:22:43 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: svn commit: r1751733 [7/18] - in /aurora/site: data/ publish/ publish/blog/ publish/blog/aurora-0-15-0-released/ publish/documentation/0.10.0/ publish/documentation/0.10.0/build-system/ publish/documentation/0.10.0/client-cluster-configuration/ publish... Date: Wed, 06 Jul 2016 21:22:40 -0000 To: commits@aurora.apache.org From: maxim@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20160706212243.2E9A63A023B@svn01-us-west.apache.org> archived-at: Wed, 06 Jul 2016 21:23:00 -0000 Added: aurora/site/publish/documentation/0.15.0/operations/configuration/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.15.0/operations/configuration/index.html?rev=1751733&view=auto ============================================================================== --- aurora/site/publish/documentation/0.15.0/operations/configuration/index.html (added) +++ aurora/site/publish/documentation/0.15.0/operations/configuration/index.html Wed Jul 6 21:22:35 2016 @@ -0,0 +1,520 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Scheduler Configuration

+ +

The Aurora scheduler can take a variety of configuration options through command-line arguments. +Examples are available under examples/scheduler/. For a list of available Aurora flags and their +documentation, see Scheduler Configuration Reference.

+ +

A Note on Configuration

+ +

Like Mesos, Aurora uses command-line flags for runtime configuration. As such the Aurora +“configuration file” is typically a scheduler.sh shell script of the form.

#!/bin/bash
+AURORA_HOME=/usr/local/aurora-scheduler
+
+# Flags controlling the JVM.
+JAVA_OPTS=(
+  -Xmx2g
+  -Xms2g
+  # GC tuning, etc.
+)
+
+# Flags controlling the scheduler.
+AURORA_FLAGS=(
+  # Port for client RPCs and the web UI
+  -http_port=8081
+  # Log configuration, etc.
+)
+
+# Environment variables controlling libmesos
+export JAVA_HOME=...
+export GLOG_v=1
+# Port used to communicate with the Mesos master and for the replicated log
+export LIBPROCESS_PORT=8083
+
+JAVA_OPTS="${JAVA_OPTS[*]}" exec "$AURORA_HOME/bin/aurora-scheduler" "${AURORA_FLAGS[@]}"
+

+ +

That way Aurora’s current flags are visible in ps and in the /vars admin endpoint.

+ +

Replicated Log Configuration

+ +

Aurora schedulers use ZooKeeper to discover log replicas and elect a leader. Only one scheduler is +leader at a given time - the other schedulers follow log writes and prepare to take over as leader +but do not communicate with the Mesos master. Either 3 or 5 schedulers are recommended in a +production deployment depending on failure tolerance and they must have persistent storage.

+ +

Below is a summary of scheduler storage configuration flags that either don’t have default values +or require attention before deploying in a production environment.

+ +

`-native_log_quorum_size`

+ +

Defines the Mesos replicated log quorum size. In a cluster with N schedulers, the flag +-native_log_quorum_size should be set to floor(N/2) + 1. So in a cluster with 1 scheduler +it should be set to 1, in a cluster with 3 it should be set to 2, and in a cluster of 5 it +should be set to 3.

+ + + + + + + + + + + + + + + + + + + + + + + +

Number of schedulers (N)	`-native_log_quorum_size` setting (`floor(N/2) + 1`)
1	1
3	2
5	3
7	4

+ +

Incorrectly setting this flag will cause data corruption to occur!

+ +

`-native_log_file_path`

+ +

Location of the Mesos replicated log files. Consider allocating a dedicated disk (preferably SSD) +for Mesos replicated log files to ensure optimal storage performance.

+ +

`-native_log_zk_group_path`

+ +

ZooKeeper path used for Mesos replicated log quorum discovery.

+ +

See code for +other available Mesos replicated log configuration options and default values.

+ +

Changing the Quorum Size

+ +

Special care needs to be taken when changing the size of the Aurora scheduler quorum. +Since Aurora uses a Mesos replicated log, similar steps need to be followed as when +changing the Mesos quorum size.

+ +

As a preparation, increase -native_log_quorum_size on each existing scheduler and restart them. +When updating from 3 to 5 schedulers, the quorum size would grow from 2 to 3.

+ +

When starting the new schedulers, use the -native_log_quorum_size set to the new value. Failing to +first increase the quorum size on running schedulers can in some cases result in corruption +or truncating of the replicated log used by Aurora. In that case, see the documentation on +recovering from backup.

+ +

Backup Configuration

+ +

Configuration options for the Aurora scheduler backup manager.

+ +

`-backup_interval`

+ +

The interval on which the scheduler writes local storage backups. The default is every hour.

+ +

`-backup_dir`

+ +

Directory to write backups to.

+ +

`-max_saved_backups`

+ +

Maximum number of backups to retain before deleting the oldest backup(s).

+ +

Process Logs

+ +

Log destination

+ +

By default, Thermos will write process stdout/stderr to log files in the sandbox. Process object configuration +allows specifying alternate log file destinations like streamed stdout/stderr or suppression of all log output. +Default behavior can be configured for the entire cluster with the following flag (through the -thermos_executor_flags +argument to the Aurora scheduler):

--runner-logger-destination=both
+

+ +

both configuration will send logs to files and stream to parent stdout/stderr outputs.

+ +

See Configuration Reference for all destination options.

+ +

Log rotation

+ +

By default, Thermos will not rotate the stdout/stderr logs from child processes and they will grow +without bound. An individual user may change this behavior via configuration on the Process object, +but it may also be desirable to change the default configuration for the entire cluster. +In order to enable rotation by default, the following flags can be applied to Thermos (through the +-thermosexecutorflags argument to the Aurora scheduler):

--runner-logger-mode=rotate
+--runner-rotate-log-size-mb=100
+--runner-rotate-log-backups=10
+

+ +

In the above example, each instance of the Thermos runner will rotate stderr/stdout logs once they +reach 100 MiB in size and keep a maximum of 10 backups. If a user has provided a custom setting for +their process, it will override these default settings.

+ +

Thermos Executor Wrapper

+ +

If you need to do computation before starting the thermos executor (for example, setting a different +--announcer-hostname parameter for every executor), then the thermos executor should be invoked + inside a wrapper script. In such a case, the aurora scheduler should be started with + -thermos_executor_path pointing to the wrapper script and -thermos_executor_resources + set to a comma separated string of all the resources that should be copied into + the sandbox (including the original thermos executor).

+ +

For example, to wrap the executor inside a simple wrapper, the scheduler will be started like this +-thermos_executor_path=/path/to/wrapper.sh -thermos_executor_resources=/usr/share/aurora/bin/thermos_executor.pex

+ +

Custom Executorâ¨â¨

+ +

If the need arises to use a Mesos executor other than the Thermos executor, the scheduler can be +configured to utilize a custom executor by specifying the -custom_executor_config flag. +The flag must be set to the path of a valid executor configuration file.â¨

+ +

The configuration file must be valid JSON and contain, at minimum, the name, command and resources fields.

+ +

executor

+ + + + + + + + + + + + + + + + + + + +

Property	Description
name (required)	Name of the executor.
command (required)	How to run the executor.
resources (required)	Overhead to use for each executor instance.

+ +

command

+ + + + + + + + + + + + + + + + + + + +

Property	Description
value (required)	The command to execute.
arguments (optional)	A list of arguments to pass to the command.
uris (optional)	List of resources to download into the task sandbox.

+ +

uris (list)

+ +

Follows the Mesos Fetcher schema

+ + + + + + + + + + + + + + + + + + + + + + + +

Property	Description
value (required)	Path to the resource needed in the sandbox.
executable (optional)	Change resource to be executable via chmod.
extract (optional)	Extract files from packed or compressed archives into the sandbox.
cache (optional)	Use caching mechanism provided by Mesos for resources.

+ +

resources (list)

+ + + + + + + + + + + + + + + + + + + +

Property	Description
name (required)	Name of the resource: cpus or mem.
type (required)	Type of resource. Should always be SCALAR.
scalar (required)	Value in float for cpus or int for mem (in MBs)

+ +

volume_mounts (list)

+ + + + + + + + + + + + + + + + + + + +

Property	Description
host_path (required)	Host path to mount inside the container.
container_path (required)	Path inside the container where `host_path` will be mounted.
mode (required)	Mode in which to mount the volume, Read-Write (RW) or Read-Only (RO).

+ +

A sample configuration is as follows:â¨ ++ { + "executor": { + "name": "myExecutor", + "command": { + "value": "myExecutor.sh", + "arguments": [ + "localhost:2181", + "-verbose", + "-config myConfiguration.config" + ], + "uris": [ + { + "value": "/dist/myExecutor.sh", + "executable": true, + "extract": false, + "cache": true + }, + { + "value": "/home/user/myConfiguration.config", + "executable": false, + "extract": false, + "cache": false + } + ] + }, + "resources": [ + { + "name": "cpus", + "type": "SCALAR", + "scalar": { + "value": 1.00 + } + }, + { + "name": "mem", + "type": "SCALAR", + "scalar": { + "value": 512 + } + } + ] + }, + "volume_mounts": [ + { + "mode": "RO", + "container_path": "/path/on/container", + "host_path": "/path/to/host/directory" + }, + { + "mode": "RW", + "container_path": "/container", + "host_path": "/host" + } + ] + } +

+ +

It should be noted that if you do not use thermos or a thermos based executor, links in the scheduler’s +Web UI for tasksâ¨ will not work (at least for the time being). +Some information about launched tasks can still be accessed via the Mesos Web UI or via the Aurora Client. +Furthermore, this configuration replaces the default thermos executor. +Work is in progress to allow support for multiple executors to co-exist within a single scheduler.

+ +

Docker containers

+ +

In order for Aurora to launch jobs using docker containers, a few extra configuration options +must be set. The docker containerizer +must be enabled on the Mesos agents by launching them with the --containerizers=docker,mesos option.

+ +

By default, Aurora will configure Mesos to copy the file specified in -thermos_executor_path +into the container’s sandbox. If using a wrapper script to launch the thermos executor, +specify the path to the wrapper in that argument. In addition, the path to the executor pex itself +must be included in the -thermos_executor_resources option. Doing so will ensure that both the +wrapper script and executor are correctly copied into the sandbox. Finally, ensure the wrapper +script does not access resources outside of the sandbox, as when the script is run from within a +docker container those resources will not exist.

+ +

A scheduler flag, -global_container_mounts allows mounting paths from the host (i.e the agent machine) +into all containers on that host. The format is a comma separated list of hostpath:containerpath[:mode] +tuples. For example -global_container_mounts=/opt/secret_keys_dir:/mnt/secret_keys_dir:ro mounts +/opt/secret_keys_dir from the agents into all launched containers. Valid modes are ro and rw.

+ +

If you would like to run a container with a read-only filesystem, it may also be necessary to +pass to use the scheduler flag -thermos_home_in_sandbox in order to set HOME to the sandbox +before the executor runs. This will make sure that the executor/runner PEX extractions happens +inside of the sandbox instead of the container filesystem root.

+ +

If you would like to supply your own parameters to docker run when launching jobs in docker +containers, you may use the following flags:

-allow_docker_parameters
+-default_docker_parameters
+

+ +

-allow_docker_parameters controls whether or not users may pass their own configuration parameters +through the job configuration files. If set to false (the default), the scheduler will reject +jobs with custom parameters. NOTE: this setting should be used with caution as it allows any job +owner to specify any parameters they wish, including those that may introduce security concerns +(privileged=true, for example).

+ +

-default_docker_parameters allows a cluster operator to specify a universal set of parameters that +should be used for every container that does not have parameters explicitly configured at the job +level. The argument accepts a multimap format:

-default_docker_parameters="read-only=true,tmpfs=/tmp,tmpfs=/run"
+

+ +

Quick Links

The ASF

© 2014-2016 Apache Software Foundation. Licensed under the Apache License v2.0. The Aurora Borealis IX photo displayed on the homepage is available under a Creative Commons BY-NC-ND 2.0 license. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.

+ + + Added: aurora/site/publish/documentation/0.15.0/operations/installation/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.15.0/operations/installation/index.html?rev=1751733&view=auto ============================================================================== --- aurora/site/publish/documentation/0.15.0/operations/installation/index.html (added) +++ aurora/site/publish/documentation/0.15.0/operations/installation/index.html Wed Jul 6 21:22:35 2016 @@ -0,0 +1,459 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Installing Aurora

+ +

Source and binary distributions can be found on our +downloads page. Installing from binary packages is +recommended for most.

+ +

Installing the scheduler
Installing worker components
Installing the client
Installing Mesos
Troubleshooting

+ +

If our binay packages don’t suite you, our package build toolchain makes it easy to build your +own packages. See the instructions to learn how.

+ +

Machine profiles

+ +

Given that many of these components communicate over the network, there are numerous ways you could +assemble them to create an Aurora cluster. The simplest way is to think in terms of three machine +profiles:

+ +

Coordinator

+ +

Components: ZooKeeper, Aurora scheduler, Mesos master

+ +

A small number of machines (typically 3 or 5) responsible for cluster orchestration. In most cases +it is fine to co-locate these components in anything but very large clusters (> 1000 machines). +Beyond that point, operators will likely want to manage these services on separate machines.

+ +

In practice, 5 coordinators have been shown to reliably manage clusters with tens of thousands of +machines.

+ +

Worker

+ +

Components: Aurora executor, Aurora observer, Mesos agent

+ +

The bulk of the cluster, where services will actually run.

+ +

Client

+ +

Components: Aurora client, Aurora admin client

+ +

Any machines that users submit jobs from.

+ +

Installing the scheduler

+ +

Ubuntu Trusty

+ +

Install Mesos +Skip down to install mesos, then run:
+
```
sudo start mesos-master
+
```
Install ZooKeeper
+
```
sudo apt-get install -y zookeeperd
+
```

Install the Aurora scheduler

sudo add-apt-repository -y ppa:openjdk-r/ppa
+sudo apt-get update
+sudo apt-get install -y openjdk-8-jre-headless wget
+
+sudo update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
+
+wget -c https://apache.bintray.com/aurora/ubuntu-trusty/aurora-scheduler_0.12.0_amd64.deb
+sudo dpkg -i aurora-scheduler_0.12.0_amd64.deb
+

+ +

CentOS 7

+ +

Install Mesos +Skip down to install mesos, then run:
+
```
sudo systemctl start mesos-master
+
```

Install ZooKeeper

sudo rpm -Uvh https://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm
+sudo yum install -y java-1.8.0-openjdk-headless zookeeper-server
+
+sudo service zookeeper-server init
+sudo systemctl start zookeeper-server
+

Install the Aurora scheduler

sudo yum install -y wget
+
+wget -c https://apache.bintray.com/aurora/centos-7/aurora-scheduler-0.12.0-1.el7.centos.aurora.x86_64.rpm
+sudo yum install -y aurora-scheduler-0.12.0-1.el7.centos.aurora.x86_64.rpm
+

+ +

Finalizing

+ +

By default, the scheduler will start in an uninitialized mode. This is because external +coordination is necessary to be certain operator error does not result in a quorum of schedulers +starting up and believing their databases are empty when in fact they should be re-joining a +cluster.

+ +

Because of this, a fresh install of the scheduler will need intervention to start up. First, +stop the scheduler service. +Ubuntu: sudo stop aurora-scheduler +CentOS: sudo systemctl stop aurora

+ +

Now initialize the database:

sudo -u aurora mkdir -p /var/lib/aurora/scheduler/db
+sudo -u aurora mesos-log initialize --path=/var/lib/aurora/scheduler/db
+

+ +

Now you can start the scheduler back up. +Ubuntu: sudo start aurora-scheduler +CentOS: sudo systemctl start aurora

+ +

Installing worker components

+ +

Ubuntu Trusty

+ +

Install Mesos +Skip down to install mesos, then run:
+
```
start mesos-slave
+
```

Install Aurora executor and observer

sudo apt-get install -y python2.7 wget
+
+# NOTE: This appears to be a missing dependency of the mesos deb package and is needed
+# for the python mesos native bindings.
+sudo apt-get -y install libcurl4-nss-dev
+
+wget -c https://apache.bintray.com/aurora/ubuntu-trusty/aurora-executor_0.12.0_amd64.deb
+sudo dpkg -i aurora-executor_0.12.0_amd64.deb
+

+ +

CentOS 7

+ +

Install Mesos +Skip down to install mesos, then run:
+
```
sudo systemctl start mesos-slave
+
```

Install Aurora executor and observer

sudo yum install -y python2 wget
+
+wget -c https://apache.bintray.com/aurora/centos-7/aurora-executor-0.12.0-1.el7.centos.aurora.x86_64.rpm
+sudo yum install -y aurora-executor-0.12.0-1.el7.centos.aurora.x86_64.rpm
+

+ +

Configuration

+ +

The executor typically does not require configuration. Command line arguments can +be passed to the executor using a command line argument on the scheduler.

+ +

The observer needs to be configured to look at the correct mesos directory in order to find task +sandboxes. You should 1st find the Mesos working directory by looking for the Mesos agent +--work_dir flag. You should see something like:

    ps -eocmd | grep "mesos-slave" | grep -v grep | tr ' ' '\n' | grep "\--work_dir"
+    --work_dir=/var/lib/mesos
+

+ +

If the flag is not set, you can view the default value like so:

    mesos-slave --help
+    Usage: mesos-slave [options]
+
+      ...
+      --work_dir=VALUE      Directory path to place framework work directories
+                            (default: /tmp/mesos)
+      ...
+

+ +

The value you find for --work_dir, /var/lib/mesos in this example, should match the Aurora +observer value for --mesos-root. You can look for that setting in a similar way on a worker +node by grepping for thermos_observer and --mesos-root. If the flag is not set, you can view +the default value like so:

    thermos_observer -h
+    Options:
+      ...
+      --mesos-root=MESOS_ROOT
+                            The mesos root directory to search for Thermos
+                            executor sandboxes [default: /var/lib/mesos]
+      ...
+

+ +

In this case the default is /var/lib/mesos and we have a match. If there is no match, you can +either adjust the mesos-master start script(s) and restart the master(s) or else adjust the +Aurora observer start scripts and restart the observers. To adjust the Aurora observer:

+ +

Ubuntu Trusty

sudo sh -c 'echo "MESOS_ROOT=/tmp/mesos" >> /etc/default/thermos'
+

+ +

NB: In Aurora releases up through 0.12.0, you’ll also need to edit /etc/init/thermos.conf like so:

diff -C 1 /etc/init/thermos.conf.orig /etc/init/thermos.conf
+*** /etc/init/thermos.conf.orig 2016-03-22 22:34:46.286199718 +0000
+--- /etc/init/thermos.conf  2016-03-22 17:09:49.357689038 +0000
+***************
+*** 24,25 ****
+--- 24,26 ----
+      --port=${OBSERVER_PORT:-1338} \
++     --mesos-root=${MESOS_ROOT:-/var/lib/mesos} \
+      --log_to_disk=NONE \
+

+ +

CentOS 7

+ +

Make an edit to add the --mesos-root flag resulting in something like:

grep -A5 OBSERVER_ARGS /etc/sysconfig/thermos
+OBSERVER_ARGS=(
+  --port=1338
+  --mesos-root=/tmp/mesos
+  --log_to_disk=NONE
+  --log_to_stderr=google:INFO
+)
+

+ +

Installing the client

+ +

Ubuntu Trusty

sudo apt-get install -y python2.7 wget
+
+wget -c https://apache.bintray.com/aurora/ubuntu-trusty/aurora-tools_0.12.0_amd64.deb
+sudo dpkg -i aurora-tools_0.12.0_amd64.deb
+

+ +

CentOS 7

sudo yum install -y python2 wget
+
+wget -c https://apache.bintray.com/aurora/centos-7/aurora-tools-0.12.0-1.el7.centos.aurora.x86_64.rpm
+sudo yum install -y aurora-tools-0.12.0-1.el7.centos.aurora.x86_64.rpm
+

+ +

Mac OS X

brew upgrade
+brew install aurora-cli
+

+ +

Configuration

+ +

Client configuration lives in a json file that describes the clusters available and how to reach +them. By default this file is at /etc/aurora/clusters.json.

+ +

Jobs may be submitted to the scheduler using the client, and are described with +job configurations expressed in .aurora files. Typically you will +maintain a single job configuration file to describe one or more deployment environments (e.g. +dev, test, prod) for a production job.

+ +

Installing Mesos

+ +

Mesos uses a single package for the Mesos master and agent. As a result, the package dependencies +are identical for both.

+ +

Mesos on Ubuntu Trusty

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF
+DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]')
+CODENAME=$(lsb_release -cs)
+
+echo "deb http://repos.mesosphere.io/${DISTRO} ${CODENAME} main" | \
+  sudo tee /etc/apt/sources.list.d/mesosphere.list
+sudo apt-get -y update
+
+# Use `apt-cache showpkg mesos | grep [version]` to find the exact version.
+sudo apt-get -y install mesos=0.25.0-0.2.70.ubuntu1404
+

+ +

Mesos on CentOS 7

sudo rpm -Uvh https://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
+sudo yum -y install mesos-0.25.0
+

+ +

Troubleshooting

+ +

So you’ve started your first cluster and are running into some issues? We’ve collected some common +stumbling blocks and solutions here to help get you moving.

+ +

Replicated log not initialized

+ +

Symptoms

+ +

Scheduler RPCs and web interface claim Storage is not READY
Scheduler log repeatedly prints messages like

  I1016 16:12:27.234133 26081 replica.cpp:638] Replica in EMPTY status
+  received a broadcasted recover request
+  I1016 16:12:27.234256 26084 recover.cpp:188] Received a recover response
+  from a replica in EMPTY status
+

+ +

Solution

+ +

When you create a new cluster, you need to inform a quorum of schedulers that they are safe to +consider their database to be empty by initializing the +replicated log. This is done to prevent the scheduler from modifying the cluster state in the event +of multiple simultaneous disk failures or, more likely, misconfiguration of the replicated log path.

+ +

Scheduler not registered

+ +

Symptoms

+ +

Scheduler log contains

Framework has not been registered within the tolerated delay.
+

+ +

Solution

+ +

Double-check that the scheduler is configured correctly to reach the Mesos master. If you are registering +the master in ZooKeeper, make sure command line argument to the master:

--zk=zk://$ZK_HOST:2181/mesos/master
+

+ +

is the same as the one on the scheduler:

-mesos_master_address=zk://$ZK_HOST:2181/mesos/master
+

+ +

Scheduler not running

+ +

Symptom

+ +

The scheduler process commits suicide regularly. This happens under error conditions, but +also on purpose in regular intervals.

+ +

Solution

+ +

Aurora is meant to be run under supervision. You have to configure a supervisor like +Monit or supervisord to run the scheduler +and restart it whenever it fails or exists on purpose.

+ +

Aurora supports an active health checking protocol on its admin HTTP interface - if a GET /health +times out or returns anything other than 200 OK the scheduler process is unhealthy and should be +restarted.

+ +

For example, monit can be configured with

if failed port 8081 send "GET /health HTTP/1.0\r\n" expect "OK\n" with timeout 2 seconds for 10 cycles then restart
+

+ +

assuming you set -http_port=8081.

+ +

Quick Links

The ASF

+ + + Added: aurora/site/publish/documentation/0.15.0/operations/monitoring/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.15.0/operations/monitoring/index.html?rev=1751733&view=auto ============================================================================== --- aurora/site/publish/documentation/0.15.0/operations/monitoring/index.html (added) +++ aurora/site/publish/documentation/0.15.0/operations/monitoring/index.html Wed Jul 6 21:22:35 2016 @@ -0,0 +1,317 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Monitoring your Aurora cluster

+ +

Before you start running important services in your Aurora cluster, it’s important to set up +monitoring and alerting of Aurora itself. Most of your monitoring can be against the scheduler, +since it will give you a global view of what’s going on.

+ +

Reading stats

+ +

The scheduler exposes a lot of instrumentation data via its HTTP interface. You can get a quick +peek at the first few of these in our vagrant image:

$ vagrant ssh -c 'curl -s localhost:8081/vars | head'
+async_tasks_completed 1004
+attribute_store_fetch_all_events 15
+attribute_store_fetch_all_events_per_sec 0.0
+attribute_store_fetch_all_nanos_per_event 0.0
+attribute_store_fetch_all_nanos_total 3048285
+attribute_store_fetch_all_nanos_total_per_sec 0.0
+attribute_store_fetch_one_events 3391
+attribute_store_fetch_one_events_per_sec 0.0
+attribute_store_fetch_one_nanos_per_event 0.0
+attribute_store_fetch_one_nanos_total 454690753
+

+ +

These values are served as Content-Type: text/plain, with each line containing a space-separated metric +name and value. Values may be integers, doubles, or strings (note: strings are static, others +may be dynamic).

+ +

If your monitoring infrastructure prefers JSON, the scheduler exports that as well:

$ vagrant ssh -c 'curl -s localhost:8081/vars.json | python -mjson.tool | head'
+{
+    "async_tasks_completed": 1009,
+    "attribute_store_fetch_all_events": 15,
+    "attribute_store_fetch_all_events_per_sec": 0.0,
+    "attribute_store_fetch_all_nanos_per_event": 0.0,
+    "attribute_store_fetch_all_nanos_total": 3048285,
+    "attribute_store_fetch_all_nanos_total_per_sec": 0.0,
+    "attribute_store_fetch_one_events": 3409,
+    "attribute_store_fetch_one_events_per_sec": 0.0,
+    "attribute_store_fetch_one_nanos_per_event": 0.0,
+

+ +

This will be the same data as above, served with Content-Type: application/json.

+ +

Viewing live stat samples on the scheduler

+ +

The scheduler uses the Twitter commons stats library, which keeps an internal time-series database +of exported variables - nearly everything in /vars is available for instant graphing. This is +useful for debugging, but is not a replacement for an external monitoring system.

+ +

You can view these graphs on a scheduler at /graphview. It supports some composition and +aggregation of values, which can be invaluable when triaging a problem. For example, if you have +the scheduler running in vagrant, check out these links: +simple graph +complex composition

+ +

Counters and gauges

+ +

Among numeric stats, there are two fundamental types of stats exported: counters and gauges. +Counters are guaranteed to be monotonically-increasing for the lifetime of a process, while gauges +may decrease in value. Aurora uses counters to represent things like the number of times an event +has occurred, and gauges to capture things like the current length of a queue. Counters are a +natural fit for accurate composition into rate ratios +(useful for sample-resistant latency calculation), while gauges are not.

+ +

Alerting

+ +

Quickstart

+ +

If you are looking for just bare-minimum alerting to get something in place quickly, set up alerting +on framework_registered and task_store_LOST. These will give you a decent picture of overall +health.

+ +

A note on thresholds

+ +

One of the most difficult things in monitoring is choosing alert thresholds. With many of these +stats, there is no value we can offer as a threshold that will be guaranteed to work for you. It +will depend on the size of your cluster, number of jobs, churn of tasks in the cluster, etc. We +recommend you start with a strict value after viewing a small amount of collected data, and then +adjust thresholds as you see fit. Feel free to ask us if you would like to validate that your alerts +and thresholds make sense.

+ +

Important stats

+ +

`jvm_uptime_secs`

+ +

Type: integer counter

+ +

The number of seconds the JVM process has been running. Comes from +RuntimeMXBean#getUptime()

+ +

Detecting resets (decreasing values) on this stat will tell you that the scheduler is failing to +stay alive.

+ +

Look at the scheduler logs to identify the reason the scheduler is exiting.

+ +

`system_load_avg`

+ +

Type: double gauge

+ +

The current load average of the system for the last minute. Comes from +OperatingSystemMXBean#getSystemLoadAverage().

+ +

A high sustained value suggests that the scheduler machine may be over-utilized.

+ +

Use standard unix tools like top and ps to track down the offending process(es).

+ +

`process_cpu_cores_utilized`

+ +

Type: double gauge

+ +

The current number of CPU cores in use by the JVM process. This should not exceed the number of +logical CPU cores on the machine. Derived from +OperatingSystemMXBean#getProcessCpuTime()

+ +

A high sustained value indicates that the scheduler is overworked. Due to current internal design +limitations, if this value is sustained at 1, there is a good chance the scheduler is under water.

+ +

There are two main inputs that tend to drive this figure: task scheduling attempts and status +updates from Mesos. You may see activity in the scheduler logs to give an indication of where +time is being spent. Beyond that, it really takes good familiarity with the code to effectively +triage this. We suggest engaging with an Aurora developer.

+ +

`task_store_LOST`

+ +

Type: integer gauge

+ +

The number of tasks stored in the scheduler that are in the LOST state, and have been rescheduled.

+ +

If this value is increasing at a high rate, it is a sign of trouble.

+ +

There are many sources of LOST tasks in Mesos: the scheduler, master, agent, and executor can all +trigger this. The first step is to look in the scheduler logs for LOST to identify where the +state changes are originating.

+ +

`scheduler_resource_offers`

+ +

Type: integer counter

+ +

The number of resource offers that the scheduler has received.

+ +

For a healthy scheduler, this value must be increasing over time.

+ +

Assuming the scheduler is up and otherwise healthy, you will want to check if the master thinks it +is sending offers. You should also look at the master’s web interface to see if it has a large +number of outstanding offers that it is waiting to be returned.

+ +

`framework_registered`

+ +

Type: binary integer counter

+ +

Will be 1 for the leading scheduler that is registered with the Mesos master, 0 for passive +schedulers,

+ +

A sustained period without a 1 (or where sum() != 1) warrants investigation.

+ +

If there is no leading scheduler, look in the scheduler and master logs for why. If there are +multiple schedulers claiming leadership, this suggests a split brain and warrants filing a critical +bug.

+ +

`rate(scheduler_log_native_append_nanos_total)/rate(scheduler_log_native_append_events)`

+ +

Type: rate ratio of integer counters

+ +

This composes two counters to compute a windowed figure for the latency of replicated log writes.

+ +

A hike in this value suggests disk bandwidth contention.

+ +

Look in scheduler logs for any reported oddness with saving to the replicated log. Also use +standard tools like vmstat and iotop to identify whether the disk has become slow or +over-utilized. We suggest using a dedicated disk for the replicated log to mitigate this.

+ +

`timed_out_tasks`

+ +

Type: integer counter

+ +

Tracks the number of times the scheduler has given up while waiting +(for -transient_task_state_timeout) to hear back about a task that is in a transient state +(e.g. ASSIGNED, KILLING), and has moved to LOST before rescheduling.

+ +

This value is currently known to increase occasionally when the scheduler fails over +(AURORA-740). However, any large spike in this +value warrants investigation.

+ +

The scheduler will log when it times out a task. You should trace the task ID of the timed out +task into the master, agent, and/or executors to determine where the message was dropped.

+ +

`http_500_responses_events`

+ +

Type: integer counter

+ +

The total number of HTTP 500 status responses sent by the scheduler. Includes API and asset serving.

+ +

An increase warrants investigation.

+ +

Look in scheduler logs to identify why the scheduler returned a 500, there should be a stack trace.

+ +

Quick Links

The ASF

+ + + Added: aurora/site/publish/documentation/0.15.0/operations/security/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.15.0/operations/security/index.html?rev=1751733&view=auto ============================================================================== --- aurora/site/publish/documentation/0.15.0/operations/security/index.html (added) +++ aurora/site/publish/documentation/0.15.0/operations/security/index.html Wed Jul 6 21:22:35 2016 @@ -0,0 +1,458 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Securing your Aurora Cluster

+ +

Aurora integrates with Apache Shiro to provide security +controls for its API. In addition to providing some useful features out of the box, Shiro +also allows Aurora cluster administrators to adapt the security system to their organizationâs +existing infrastructure. The announcer in the Aurora thermos executor also supports security +controls for talking to ZooKeeper.

+ +

Enabling Security
Authentication + +
- HTTP Basic Authentication + +
  - Server Configuration
  - Client Configuration
- HTTP SPNEGO Authentication (Kerberos) + +
  - Server Configuration
  - Client Configuration
Authorization + +
- Using an INI file to define security controls + +
  - Caveats
Implementing a Custom Realm + +
- Packaging a realm module
Known Issues
Announcer Authentication + +
- ZooKeeper authentication configuration
- Executor settings

+ +

Enabling Security

+ +

There are two major components of security: +authentication and authorization. A +cluster administrator may choose the approach used for each, and may also implement custom +mechanisms for either. Later sections describe the options available. To enable authentication + for the announcer, see Announcer Authentication

+ +

Authentication

+ +

The scheduler must be configured with instructions for how to process authentication +credentials at a minimum. There are currently two built-in authentication schemes - +HTTP Basic Authentication, and +SPNEGO (Kerberos).

+ +

HTTP Basic Authentication

+ +

Basic Authentication is a very quick way to add some security. It is supported +by all major browsers and HTTP client libraries with minimal work. However, +before relying on Basic Authentication you should be aware of the security +considerations.

+ +

Server Configuration

+ +

At a minimum you need to set 4 command-line flags on the scheduler:

-http_authentication_mechanism=BASIC
+-shiro_realm_modules=INI_AUTHNZ
+-shiro_ini_path=path/to/security.ini
+

+ +

And create a security.ini file like so:

[users]
+sally = apple, admin
+
+[roles]
+admin = *
+

+ +

The details of the security.ini file are explained below. Note that this file contains plaintext, +unhashed passwords.

+ +

Client Configuration

+ +

To configure the client for HTTP Basic authentication, add an entry to ~/.netrc with your credentials

% cat ~/.netrc
+# ...
+
+machine aurora.example.com
+login sally
+password apple
+
+# ...
+

+ +

No changes are required to clusters.json.

+ +

HTTP SPNEGO Authentication (Kerberos)

+ +

Server Configuration

+ +

At a minimum you need to set 6 command-line flags on the scheduler:

-http_authentication_mechanism=NEGOTIATE
+-shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ
+-kerberos_server_principal=HTTP/aurora.example.com@EXAMPLE.COM
+-kerberos_server_keytab=path/to/aurora.example.com.keytab
+-shiro_ini_path=path/to/security.ini
+

+ +

And create a security.ini file like so:

% cat path/to/security.ini
+[users]
+sally = _, admin
+
+[roles]
+admin = *
+

+ +

What’s going on here? First, Aurora must be configured to request Kerberos credentials when presented with an +unauthenticated request. This is achieved by setting

-http_authentication_mechanism=NEGOTIATE
+

+ +

Next, a Realm module must be configured to authenticate the current request using the Kerberos +credentials that were requested. Aurora ships with a realm module that can do this

-shiro_realm_modules=KERBEROS5_AUTHN[,...]
+

+ +

The Kerberos5Realm requires a keytab file and a server principal name. The principal name will usually +be in the form HTTP/aurora.example.com@EXAMPLE.COM.

-kerberos_server_principal=HTTP/aurora.example.com@EXAMPLE.COM
+-kerberos_server_keytab=path/to/aurora.example.com.keytab
+

+ +

The Kerberos5 realm module is authentication-only. For scheduler security to work you must also +enable a realm module that provides an Authorizer implementation. For example, to do this using the +IniShiroRealmModule:

-shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ
+

+ +

You can then configure authorization using a security.ini file as described below +(the password field is ignored). You must configure the realm module with the path to this file:

-shiro_ini_path=path/to/security.ini
+

+ +

Client Configuration

+ +

To use Kerberos on the client-side you must build Kerberos-enabled client binaries. Do this with

./pants binary src/main/python/apache/aurora/kerberos:kaurora
+./pants binary src/main/python/apache/aurora/kerberos:kaurora_admin
+

+ +

You must also configure each cluster where you’ve enabled Kerberos on the scheduler +to use Kerberos authentication. Do this by setting auth_mechanism to KERBEROS +in clusters.json.

% cat ~/.aurora/clusters.json
+{
+    "devcluser": {
+        "auth_mechanism": "KERBEROS",
+        ...
+    },
+    ...
+}
+

+ +

Authorization

+ +

Given a means to authenticate the entity a client claims they are, we need to define what privileges they have.

+ +

Using an INI file to define security controls

+ +

The simplest security configuration for Aurora is an INI file on the scheduler. For small +clusters, or clusters where the users and access controls change relatively infrequently, this is +likely the preferred approach. However you may want to avoid this approach if access permissions +are rapidly changing, or if your access control information already exists in another system.

+ +

You can enable INI-based configuration with following scheduler command line arguments:

-http_authentication_mechanism=BASIC
+-shiro_ini_path=path/to/security.ini
+

+ +

note As the argument name reveals, this is using Shiroâs +IniRealm behind +the scenes.

+ +

The INI file will contain two sections - users and roles. Hereâs an example for what might +be in security.ini:

[users]
+sally = apple, admin
+jim = 123456, accounting
+becky = letmein, webapp
+larry = 654321,accounting
+steve = password
+
+[roles]
+admin = *
+accounting = thrift.AuroraAdmin:setQuota
+webapp = thrift.AuroraSchedulerManager:*:webapp
+

+ +

The users section defines user user credentials and the role(s) they are members of. These lines +are of the format <user> = <password>[, <role>...]. As you probably noticed, the passwords are +in plaintext and as a result read access to this file should be restricted.

+ +

In this configuration, each user has different privileges for actions in the cluster because +of the roles they are a part of:

+ +

admin is granted all privileges
accounting may adjust the amount of resource quota for any role
webapp represents a collection of jobs that represents a service, and its members may create and modify any jobs owned by it

+ +

Caveats

+ +

You might find documentation on the Internet suggesting there are additional sections in shiro.ini, +like [main] and [urls]. These are not supported by Aurora as it uses a different mechanism to configure +those parts of Shiro. Think of Aurora’s security.ini as a subset with only [users] and [roles] sections.

+ +

Implementing Delegated Authorization

+ +

It is possible to leverage Shiro’s runAs feature by implementing a custom Servlet Filter that provides +the capability and passing it’s fully qualified class name to the command line argument +-shiro_after_auth_filter. The filter is registered in the same filter chain as the Shiro auth filters +and is placed after the Shiro auth filters in the filter chain. This ensures that the Filter is invoked +after the Shiro filters have had a chance to authenticate the request.

+ +

Implementing a Custom Realm

+ +

Since Auroraâs security is backed by Apache Shiro, you can implement a +custom Realm to define organization-specific security behavior.

+ +

In addition to using Shiro’s standard APIs to implement a Realm you can link against Aurora to +access the type-safe Permissions Aurora uses. See the Javadoc for org.apache.aurora.scheduler.spi +for more information.

+ +

Packaging a realm module

+ +

Package your custom Realm(s) with a Guice module that exposes a Set<Realm> multibinding.

package com.example;
+
+import com.google.inject.AbstractModule;
+import com.google.inject.multibindings.Multibinder;
+import org.apache.shiro.realm.Realm;
+
+public class MyRealmModule extends AbstractModule {
+  @Override
+  public void configure() {
+    Realm myRealm = new MyRealm();
+
+    Multibinder.newSetBinder(binder(), Realm.class).addBinding().toInstance(myRealm);
+  }
+
+  static class MyRealm implements Realm {
+    // Realm implementation.
+  }
+}
+

+ +

To use your module in the scheduler, include it as a realm module based on its fully-qualified +class name:

-shiro_realm_modules=KERBEROS5_AUTHN,INI_AUTHNZ,com.example.MyRealmModule
+

+ +

Known Issues

+ +

While the APIs and SPIs we ship with are stable as of 0.8.0, we are aware of several incremental +improvements. Please follow, vote, or send patches.

+ +

Relevant tickets: +* AURORA-343: HTTPS support +* AURORA-1248: Client retries 4xx errors +* AURORA-1279: Remove kerberos-specific build targets +* AURORA-1293: Consider defining a JSON format in place of INI +* AURORA-1179: Supported hashed passwords in security.ini +* AURORA-1295: Support security for the ReadOnlyScheduler service

+ +

Announcer Authentication

+ +

The Thermos executor can be configured to authenticate with ZooKeeper and include +an ACL +on the nodes it creates, which will specify +the privileges of clients to perform different actions on these nodes. This +feature is enabled by specifying an ACL configuration file to the executor with the +--announcer-zookeeper-auth-config command line argument.

+ +

When this feature is not enabled, nodes created by the executor will have ‘world/all’ permission +(ZOO_OPEN_ACL_UNSAFE). In most production environments, operators should specify an ACL and +limit access.

+ +

ZooKeeper Authentication Configuration

+ +

The configuration file must be formatted as JSON with the following schema:

{
+  "auth": [
+    {
+      "scheme": "<scheme>",
+      "credential": "<plain_credential>"
+    }
+  ],
+  "acl": [
+    {
+      "scheme": "<scheme>",
+      "credential": "<plain_credential>",
+      "permissions": {
+        "read": <bool>,
+        "write": <bool>,
+        "create": <bool>,
+        "delete": <bool>,
+        "admin": <bool>
+      }
+    }
+  ]
+}
+

+ +

The scheme +defines the encoding of the credential field. Note that these fields are passed directly to +ZooKeeper (except in the case of digest scheme, where the executor will hash and encode +the credential appropriately before passing it to ZooKeeper). In addition to acl, a list of +authentication credentials must be provided in auth to use for the connection.

+ +

All properties of the permissions object will default to False if not provided.

+ +

Executor settings

+ +

To enable the executor to authenticate against ZK, --announcer-zookeeper-auth-config should be +set to the configuration file.

+ +

Quick Links

The ASF

+ + +

Documentation + + + 0.15.0 + (latest) + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Scheduler Configuration

A Note on Configuration

Replicated Log Configuration

-native_log_quorum_size

-native_log_file_path

-native_log_zk_group_path

Changing the Quorum Size

Backup Configuration

-backup_interval

-backup_dir

-max_saved_backups

Process Logs

Log destination

Log rotation

Thermos Executor Wrapper

Custom Executorâ¨â¨

executor

command

uris (list)

resources (list)

volume_mounts (list)

Docker containers

Quick Links

The ASF

Documentation + + + 0.15.0 + (latest) + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Installing Aurora

Machine profiles

Coordinator

Worker

Client

Installing the scheduler

Ubuntu Trusty

CentOS 7

Finalizing

Installing worker components

Ubuntu Trusty

CentOS 7

Configuration

Ubuntu Trusty

CentOS 7

Installing the client

Ubuntu Trusty

CentOS 7

Mac OS X

Configuration

Installing Mesos

Mesos on Ubuntu Trusty

Mesos on CentOS 7

Troubleshooting

Replicated log not initialized

Symptoms

Solution

Scheduler not registered

Symptoms

Solution

Scheduler not running

Symptom

Solution

Quick Links

The ASF

Documentation + + + 0.15.0 + (latest) + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Monitoring your Aurora cluster

Reading stats

Viewing live stat samples on the scheduler

Counters and gauges

Alerting

Quickstart

A note on thresholds

Important stats

jvm_uptime_secs

system_load_avg

process_cpu_cores_utilized

task_store_LOST

scheduler_resource_offers

framework_registered

rate(scheduler_log_native_append_nanos_total)/rate(scheduler_log_native_append_events)

timed_out_tasks

http_500_responses_events

Quick Links

Documentation + +

`-native_log_quorum_size`

`-native_log_file_path`

`-native_log_zk_group_path`

`-backup_interval`

`-backup_dir`

`-max_saved_backups`

Custom Executorâ¨â¨

Documentation + +

Documentation + +

`jvm_uptime_secs`

`system_load_avg`

`process_cpu_cores_utilized`

`task_store_LOST`

`scheduler_resource_offers`

`framework_registered`

`rate(scheduler_log_native_append_nanos_total)/rate(scheduler_log_native_append_events)`

`timed_out_tasks`

`http_500_responses_events`

Documentation + +