; Tue, 14 Jun 2016 21:35:33 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1748470 [8/19] - in /aurora/site: data/ publish/ publish/blog/ publish/blog/aurora-0-14-0-released/ publish/documentation/0.10.0/ publish/documentation/0.10.0/build-system/ publish/documentation/0.10.0/client-cluster-configuration/ publish... Date: Tue, 14 Jun 2016 21:35:30 -0000 To: commits@aurora.apache.org From: serb@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20160614213533.652153A039A@svn01-us-west.apache.org> archived-at: Tue, 14 Jun 2016 21:35:51 -0000 Added: aurora/site/publish/documentation/0.14.0/operations/storage/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/operations/storage/index.html?rev=1748470&view=auto ============================================================================== --- aurora/site/publish/documentation/0.14.0/operations/storage/index.html (added) +++ aurora/site/publish/documentation/0.14.0/operations/storage/index.html Tue Jun 14 21:35:25 2016 @@ -0,0 +1,222 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Aurora Scheduler Storage

+ +

Overview
Replicated Log Configuration
Backup Configuration
Storage Semantics + +

+ +

Overview

+ +

Aurora scheduler maintains data that need to be persisted to survive failovers and restarts. +For example:

+ +

Task configurations and scheduled task instances
Job update configurations and update progress
Production resource quotas
Mesos resource offer host attributes

+ +

Aurora solves its persistence needs by leveraging the Mesos implementation of a Paxos replicated +log [1] +[2] with a key-value +LevelDB storage as persistence media.

+ +

Conceptually, it can be represented by the following major components:

+ +

Volatile storage: in-memory cache of all available data. Implemented via in-memory +H2 Database and accessed via +MyBatis.
Log manager: interface between Aurora storage and Mesos replicated log. The default schema format +is thrift. Data is stored in serialized binary form.
Snapshot manager: all data is periodically persisted in Mesos replicated log in a single snapshot. +This helps establishing periodic recovery checkpoints and speeds up volatile storage recovery on +restart.
Backup manager: as a precaution, snapshots are periodically written out into backup files. +This solves a disaster recovery problem +in case of a complete loss or corruption of Mesos log files.

+ +

Storage hierarchy

+ +

Storage Semantics

+ +

Implementation details of the Aurora storage system. Understanding those can sometimes be useful +when investigating performance issues.

+ +

Reads, writes, modifications

+ +

All services in Aurora access data via a set of predefined store interfaces (aka stores) logically +grouped by the type of data they serve. Every interface defines a specific set of operations allowed +on the data thus abstracting out the storage access and the actual persistence implementation. The +latter is especially important in view of a general immutability of persisted data. With the Mesos +replicated log as the underlying persistence solution, data can be read and written easily but not +modified. All modifications are simulated by saving new versions of modified objects. This feature +and general performance considerations justify the existence of the volatile in-memory store.

+ +

Read lifecycle

+ +

There are two types of reads available in Aurora: consistent and weakly-consistent. The difference +is explained below.

+ +

All reads are served from the volatile storage making reads generally cheap storage operations +from the performance standpoint. The majority of the volatile stores are represented by the +in-memory H2 database. This allows for rich schema definitions, queries and relationships that +key-value storage is unable to match.

+ +

Write lifecycle

+ +

Writes are more involved operations since in addition to updating the volatile store data has to be +appended to the replicated log. Data is not available for reads until fully ack-ed by both +replicated log and volatile storage.

+ +

Atomicity, consistency and isolation

+ +

Aurora uses write-ahead logging to ensure +consistency between replicated and volatile storage. In Aurora, data is first written into the +replicated log and only then updated in the volatile store.

+ +

Aurora storage uses read-write locks to serialize data mutations and provide consistent view of the +available data. The available Storage interface exposes 3 major types of operations: +* consistentRead - access is locked using reader’s lock and provides consistent view on read +* weaklyConsistentRead - access is lock-less. Delivers best contention performance but may result +in stale reads +* write - access is fully serialized by using writer’s lock. Operation success requires both +volatile and replicated writes to succeed.

+ +

The consistency of the volatile store is enforced via H2 transactional isolation.

+ +

Population on restart

+ +

Any time a scheduler restarts, it restores its volatile state from the most recent position recorded +in the replicated log by restoring the snapshot and replaying individual log entries on top to fully +recover the state up to the last write.

+ +

Quick Links

The ASF

Copyright 2014 Apache Software Foundation. Licensed under the Apache License v2.0. The Aurora Borealis IX photo displayed on the homepage is available under a Creative Commons BY-NC-ND 2.0 license. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.

+ + + Propchange: aurora/site/publish/documentation/0.14.0/operations/storage/index.html ------------------------------------------------------------------------------ svn:eol-style = native Propchange: aurora/site/publish/documentation/0.14.0/operations/storage/index.html ------------------------------------------------------------------------------ svn:keywords = Id Propchange: aurora/site/publish/documentation/0.14.0/operations/storage/index.html ------------------------------------------------------------------------------ svn:mime-type = text/html Added: aurora/site/publish/documentation/0.14.0/reference/client-cluster-configuration/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/reference/client-cluster-configuration/index.html?rev=1748470&view=auto ============================================================================== --- aurora/site/publish/documentation/0.14.0/reference/client-cluster-configuration/index.html (added) +++ aurora/site/publish/documentation/0.14.0/reference/client-cluster-configuration/index.html Tue Jun 14 21:35:25 2016 @@ -0,0 +1,250 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Client Cluster Configuration

+ +

A cluster configuration file is used by the Aurora client to describe the Aurora clusters with +which it can communicate. Ultimately this allows client users to reference clusters with short names +like us-east and eu.

+ +

A cluster configuration is formatted as JSON. The simplest cluster configuration is one that +communicates with a single (non-leader-elected) scheduler. For example:

[{
+  "name": "example",
+  "scheduler_uri": "http://localhost:55555",
+}]
+

+ +

A configuration for a leader-elected scheduler would contain something like:

[{
+  "name": "example",
+  "zk": "192.168.33.7",
+  "scheduler_zk_path": "/aurora/scheduler"
+}]
+

+ +

The following properties may be set:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Property	Type	Description
name	String	Cluster name (Required)
slave_root	String	Path to Mesos agent work dir (Required)
slaverundirectory	String	Name of Mesos agent run dir (Required)
zk	String	Hostname of ZooKeeper instance used to resolve Aurora schedulers.
zk_port	Integer	Port of ZooKeeper instance used to locate Aurora schedulers (Default: 2181)
schedulerzkpath	String	ZooKeeper path under which scheduler instances are registered.
scheduler_uri	String	URI of Aurora scheduler instance.
proxy_url	String	Used by the client to format URLs for display.
auth_mechanism	String	The authentication mechanism to use when communicating with the scheduler. (Default: UNAUTHENTICATED)

+ +

Details

+ +

`name`

+ +

The name of the Aurora cluster represented by this entry. This name will be the cluster portion of +any job keys identifying jobs running within the cluster.

+ +

`slave_root`

+ +

The path on the Mesos agents where executing tasks can be found. It is used in combination with the +slave_run_directory property by aurora task run and aurora task ssh to change into the sandbox +directory after connecting to the host. This value should match the value passed to mesos-slave +as -work_dir.

+ +

`slave_run_directory`

+ +

The name of the directory where the task run can be found. This is used in combination with the +slave_root property by aurora task run and aurora task ssh to change into the sandbox +directory after connecting to the host. This should almost always be set to latest.

+ +

`zk`

+ +

The hostname of the ZooKeeper instance used to resolve the Aurora scheduler. Aurora uses ZooKeeper +to elect a leader. The client will connect to this ZooKeeper instance to determine the current +leader. This host should match the host passed to the scheduler as -zk_endpoints.

+ +

`zk_port`

+ +

The port on which the ZooKeeper instance is running. If not set this will default to the standard +ZooKeeper port of 2181. This port should match the port in the host passed to the scheduler as +-zk_endpoints.

+ +

`scheduler_zk_path`

+ +

The path on the ZooKeeper instance under which the Aurora serverset is registered. This value should +match the value passed to the scheduler as -serverset_path.

+ +

`scheduler_uri`

+ +

The URI of the scheduler. This would be used in place of the ZooKeeper related configuration above +in circumstances where direct communication with a single scheduler is needed (e.g. testing +environments). It is strongly advised to never use this property for production deploys.

+ +

`proxy_url`

+ +

Instead of using the hostname of the leading scheduler as the base url, if proxy_url is set, its +value will be used instead. In that scenario the value for proxy_url would be, for example, the +URL of your VIP in a loadbalancer or a roundrobin DNS name.

+ +

`auth_mechanism`

+ +

The identifier of an authentication mechanism that the client should use when communicating with the +scheduler. Support for values other than UNAUTHENTICATED requires a matching scheduler-side +security configuration.

+ +

Quick Links

The ASF

+ + + Propchange: aurora/site/publish/documentation/0.14.0/reference/client-cluster-configuration/index.html ------------------------------------------------------------------------------ svn:eol-style = native Propchange: aurora/site/publish/documentation/0.14.0/reference/client-cluster-configuration/index.html ------------------------------------------------------------------------------ svn:keywords = Id Propchange: aurora/site/publish/documentation/0.14.0/reference/client-cluster-configuration/index.html ------------------------------------------------------------------------------ svn:mime-type = text/html Added: aurora/site/publish/documentation/0.14.0/reference/client-commands/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/reference/client-commands/index.html?rev=1748470&view=auto ============================================================================== --- aurora/site/publish/documentation/0.14.0/reference/client-commands/index.html (added) +++ aurora/site/publish/documentation/0.14.0/reference/client-commands/index.html Tue Jun 14 21:35:25 2016 @@ -0,0 +1,448 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Aurora Client Commands

+ +

Introduction
Cluster Configuration
Job Keys
Modifying Aurora Client Commands
Regular Jobs + +
- Creating and Running a Job
- Running a Command On a Running Job
- Killing a Job
- Adding Instances
- Updating a Job + +
  - Coordinated job updates
- Renaming a Job
- Restarting Jobs
Cron Jobs
Comparing Jobs
Viewing/Examining Jobs + +

+ +

Introduction

+ +

Once you have written an .aurora configuration file that describes +your Job and its parameters and functionality, you interact with Aurora +using Aurora Client commands. This document describes all of these commands +and how and when to use them. All Aurora Client commands start with +aurora, followed by the name of the specific command and its +arguments.

+ +

Job keys are a very common argument to Aurora commands, as well as the +gateway to useful information about a Job. Before using Aurora, you +should read the next section which describes them in detail. The section +after that briefly describes how you can modify the behavior of certain +Aurora Client commands, linking to a detailed document about how to do +that.

+ +

This is followed by the Regular Jobs section, which describes the basic +Client commands for creating, running, and manipulating Aurora Jobs. +After that are sections on Comparing Jobs and Viewing/Examining Jobs. In +other words, various commands for getting information and metadata about +Aurora Jobs.

+ +

Cluster Configuration

+ +

The client must be able to find a configuration file that specifies available clusters. This file +declares shorthand names for clusters, which are in turn referenced by job configuration files +and client commands.

+ +

The client will load at most two configuration files, making both of their defined clusters +available. The first is intended to be a system-installed cluster, using the path specified in +the environment variable AURORA_CONFIG_ROOT, defaulting to /etc/aurora/clusters.json if the +environment variable is not set. The second is a user-installed file, located at +~/.aurora/clusters.json.

+ +

For more details on cluster configuration see the +Client Cluster Configuration documentation.

+ +

Job Keys

+ +

A job key is a unique system-wide identifier for an Aurora-managed +Job, for example cluster1/web-team/test/experiment204. It is a 4-tuple +consisting of, in order, cluster, role, environment, and +jobname, separated by /s. Cluster is the name of an Aurora +cluster. Role is the Unix service account under which the Job +runs. Environment is a namespace component like devel, test, +prod, or stagingN. Jobname is the Job’s name.

+ +

The combination of all four values uniquely specifies the Job. If any +one value is different from that of another job key, the two job keys +refer to different Jobs. For example, job key +cluster1/tyg/prod/workhorse is different from +cluster1/tyg/prod/workcamel is different from +cluster2/tyg/prod/workhorse is different from +cluster2/foo/prod/workhorse is different from +cluster1/tyg/test/workhorse.

+ +

Role names are user accounts existing on the agent machines. If you don’t know what accounts +are available, contact your sysadmin.

+ +

Environment names are namespaces; you can count on prod, devel and test existing.

+ +

Modifying Aurora Client Commands

+ +

For certain Aurora Client commands, you can define hook methods that run +either before or after an action that takes place during the command’s +execution, as well as based on whether the action finished successfully or failed +during execution. Basically, a hook is code that lets you extend the +command’s actions. The hook executes on the client side, specifically on +the machine executing Aurora commands.

+ +

Hooks can be associated with these Aurora Client commands.

+ +

job create
job kill
job restart

+ +

The process for writing and activating them is complex enough +that we explain it in a devoted document, Hooks for Aurora Client API.

+ +

Regular Jobs

+ +

This section covers Aurora commands related to running, killing, +renaming, updating, and restarting a basic Aurora Job.

+ +

Creating and Running a Job

aurora job create <job key> <configuration file>
+

+ +

Creates and then runs a Job with the specified job key based on a .aurora configuration file. +The configuration file may also contain and activate hook definitions.

+ +

Running a Command On a Running Job

aurora task run CLUSTER/ROLE/ENV/NAME[/INSTANCES] <cmd>
+

+ +

Runs a shell command on all machines currently hosting shards of a +single Job.

+ +

run supports the same command line wildcards used to populate a Job’s +commands; i.e. anything in the {{mesos.*}} and {{thermos.*}} +namespaces.

+ +

Killing a Job

aurora job killall CLUSTER/ROLE/ENV/NAME
+

+ +

Kills all Tasks associated with the specified Job, blocking until all +are terminated. Defaults to killing all instances in the Job.

+ +

The <configuration file> argument for kill is optional. Use it only +if it contains hook definitions and activations that affect the +kill command.

+ +

Adding Instances

aurora job add CLUSTER/ROLE/ENV/NAME/INSTANCE <count>
+

+ +

Adds <count> instances to the existing job. The configuration of the new instances is derived from +an active job instance pointed by the /INSTANCE part of the job specification. This command is +a simpler way to scale out an existing job when an instance with desired task configuration +already exists. Use aurora update start to add instances with a new (updated) configuration.

+ +

Updating a Job

+ +

You can manage job updates using the aurora update command. Please see +the Job Update documentation for more details.

+ +

Renaming a Job

+ +

Renaming is a tricky operation as downstream clients must be informed of +the new name. A conservative approach +to renaming suitable for production services is:

+ +

Modify the Aurora configuration file to change the role, +environment, and/or name as appropriate to the standardized naming +scheme.
Check that only these naming components have changed +with aurora diff.
+
```
aurora job diff CLUSTER/ROLE/ENV/NAME <job_configuration>
+
```
Create the (identical) job at the new key. You may need to request a +temporary quota increase.
+
```
aurora job create CLUSTER/ROLE/ENV/NEW_NAME <job_configuration>
+
```
Migrate all clients over to the new job key. Update all links and +dashboards. Ensure that both job keys run identical versions of the +code while in this state.
After verifying that all clients have successfully moved over, kill +the old job.
+
```
aurora job killall CLUSTER/ROLE/ENV/NAME
+
```
If you received a temporary quota increase, be sure to let the +powers that be know you no longer need the additional capacity.

+ +

Restarting Jobs

+ +

restart restarts all of a job key identified Job’s shards:

aurora job restart CLUSTER/ROLE/ENV/NAME[/INSTANCES]
+

+ +

Restarts are controlled on the client side, so aborting +the job restart command halts the restart operation.

+ +

Note: job restart only applies its command line arguments and does not +use or is affected by update.config. Restarting +does not involve a configuration change. To update the +configuration, use update.config.

+ +

The --config argument for restart is optional. Use it only +if it contains hook definitions and activations that affect the +job restart command.

+ +

Cron Jobs

+ +

You can manage cron jobs using the aurora cron command. Please see +the Cron Jobs Feature for more details.

+ +

Comparing Jobs

aurora job diff CLUSTER/ROLE/ENV/NAME <job configuration>
+

+ +

Compares a job configuration against a running job. By default the diff +is determined using diff, though you may choose an alternate + diff program by specifying the DIFF_VIEWER environment variable.

+ +

Viewing/Examining Jobs

+ +

Above we discussed creating, killing, and updating Jobs. Here we discuss +how to view and examine Jobs.

+ +

Listing Jobs

aurora config list <job configuration>
+

+ +

Lists all Jobs registered with the Aurora scheduler in the named cluster for the named role.

+ +

Inspecting a Job

aurora job inspect CLUSTER/ROLE/ENV/NAME <job configuration>
+

+ +

inspect verifies that its specified job can be parsed from a +configuration file, and displays the parsed configuration.

+ +

Checking Your Quota

aurora quota get CLUSTER/ROLE
+

+ +

Prints the production quota allocated to the role’s value at the given +cluster. Only non-dedicated +production jobs consume quota.

+ +

Finding a Job on Web UI

+ +

When you create a job, part of the output response contains a URL that goes +to the job’s scheduler UI page. For example:

vagrant@precise64:~$ aurora job create devcluster/www-data/prod/hello /vagrant/examples/jobs/hello_world.aurora
+INFO] Creating job hello
+INFO] Response from scheduler: OK (message: 1 new tasks pending for job www-data/prod/hello)
+INFO] Job url: http://precise64:8081/scheduler/www-data/prod/hello
+

+ +

You can go to the scheduler UI page for this job via http://precise64:8081/scheduler/www-data/prod/hello +You can go to the overall scheduler UI page by going to the part of that URL that ends at scheduler; http://precise64:8081/scheduler

+ +

Once you click through to a role page, you see Jobs arranged +separately by pending jobs, active jobs and finished jobs. +Jobs are arranged by role, typically a service account for +production jobs and user accounts for test or development jobs.

+ +

Getting Job Status

aurora job status <job_key>
+

+ +

Returns the status of recent tasks associated with the +job_key specified Job in its supplied cluster. Typically this includes +a mix of active tasks (running or assigned) and inactive tasks +(successful, failed, and lost.)

+ +

Opening the Web UI

+ +

Use the Job’s web UI scheduler URL or the aurora status command to find out on which +machines individual tasks are scheduled. You can open the web UI via the +open command line command if invoked from your machine:

aurora job open [<cluster>[/<role>[/<env>/<job_name>]]]
+

+ +

If only the cluster is specified, it goes directly to that cluster’s +scheduler main page. If the role is specified, it goes to the top-level +role page. If the full job key is specified, it goes directly to the job +page where you can inspect individual tasks.

+ +

SSHing to a Specific Task Machine

aurora task ssh <job_key> <shard number>
+

+ +

You can have the Aurora client ssh directly to the machine that has been +assigned a particular Job/shard number. This may be useful for quickly +diagnosing issues such as performance issues or abnormal behavior on a +particular machine.

+ +

Templating Command Arguments

aurora task run [-e] [-t THREADS] <job_key> -- <<command-line>>
+

+ +

Given a job specification, run the supplied command on all hosts and +return the output. You may use the standard Mustache templating rules:

+ +

{{thermos.ports[name]}} substitutes the specific named port of the +task assigned to this machine
{{mesos.instance}} substitutes the shard id of the job’s task +assigned to this machine
{{thermos.task_id}} substitutes the task id of the job’s task +assigned to this machine

+ +

For example, the following type of pattern can be a powerful diagnostic +tool:

aurora task run -t5 cluster1/tyg/devel/seizure -- \
+  'curl -s -m1 localhost:{{thermos.ports[http]}}/vars | grep uptime'
+

+ +

By default, the command runs in the Task’s sandbox. The -e option can +run the command in the executor’s sandbox. This is mostly useful for +Aurora administrators.

+ +

You can parallelize the runs by using the -t option.

+ +

Quick Links

The ASF

+ + + Propchange: aurora/site/publish/documentation/0.14.0/reference/client-commands/index.html ------------------------------------------------------------------------------ svn:eol-style = native Propchange: aurora/site/publish/documentation/0.14.0/reference/client-commands/index.html ------------------------------------------------------------------------------ svn:keywords = Id Propchange: aurora/site/publish/documentation/0.14.0/reference/client-commands/index.html ------------------------------------------------------------------------------ svn:mime-type = text/html Added: aurora/site/publish/documentation/0.14.0/reference/client-hooks/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/reference/client-hooks/index.html?rev=1748470&view=auto ============================================================================== --- aurora/site/publish/documentation/0.14.0/reference/client-hooks/index.html (added) +++ aurora/site/publish/documentation/0.14.0/reference/client-hooks/index.html Tue Jun 14 21:35:25 2016 @@ -0,0 +1,379 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Hooks for Aurora Client API

+ +

You can execute hook methods around Aurora API Client methods when they are called by the Aurora Command Line commands.

+ +

Explaining how hooks work is a bit tricky because of some indirection about what they apply to. Basically, a hook is code that executes when a particular Aurora Client API method runs, letting you extend the method’s actions. The hook executes on the client side, specifically on the machine executing Aurora commands.

+ +

The catch is that hooks are associated with Aurora Client API methods, which users don’t directly call. Instead, users call Aurora Command Line commands, which call Client API methods during their execution. Since which hooks run depend on which Client API methods get called, you will need to know which Command Line commands call which API methods. Later on, there is a table showing the various associations.

+ +

Terminology Note: From now on, “method(s)” refer to Client API methods, and “command(s)” refer to Command Line commands.

+ +

Hook Types
Execution Order
Hookable Methods
Activating and Using Hooks
.aurora Config File Settings
Command Line
Hooks Protocol + +
- pre_ Methods
- err_ Methods
- post_ Methods
Generic Hooks
Hooks Process Checklist

+ +

Hook Types

+ +

Hooks have three basic types, differing by when they run with respect to their associated method.

+ +

pre_<method_name>: When its associated method is called, the pre_ hook executes first, then the called method. If the pre_ hook fails, the method never runs. Later code that expected the method to succeed may be affected by this, and result in terminating the Aurora client.

+ +

Note that a pre_ hook can error-trap internally so it does not +return False. Designers/contributors of new pre_ hooks should +consider whether or not to error-trap them. You can error trap at the +highest level very generally and always pass the pre_ hook by +returning True. For example:

def pre_create(...):
+  do_something()  # if do_something fails with an exception, the create_job is not attempted!
+  return True
+
+# However...
+def pre_create(...):
+  try:
+    do_something()  # may cause exception
+  except Exception:  # generic error trap will catch it
+    pass  # and ignore the exception
+  return True  # create_job will run in any case!
+

+ +

post_<method_name>: A post_ hook executes after its associated method successfully finishes running. If it fails, the already executed method is unaffected. A post_ hook’s error is trapped, and any later operations are unaffected.

+ +

err_<method_name>: Executes only when its associated method returns a status other than OK or throws an exception. If an err_ hook fails, the already executed method is unaffected. An err_ hook’s error is trapped, and any later operations are unaffected.

+ +

Execution Order

+ +

A command with pre_, post_, and err_ hooks defined and activated for its called method executes in the following order when the method successfully executes:

+ +

Command called
Command code executes
Method Called
pre_ method hook runs
Method runs and successfully finishes
post_ method hook runs
Command code executes
Command execution ends

+ +

The following is what happens when, for the same command and hooks, the method associated with the command suffers an error and does not successfully finish executing:

+ +

Command called
Command code executes
Method Called
pre_ method hook runs
Method runs and fails
err_ method hook runs
Command Code executes (if err_ method does not end the command execution)
Command execution ends

+ +

Note that the post_ and err_ hooks for the same method can never both run for a single execution of that method.

+ +

Hookable Methods

+ +

You can associate pre_, post_, and err_ hooks with the following methods. Since you do not directly interact with the methods, but rather the Aurora Command Line commands that call them, for each method we also list the command(s) that can call the method. Note that a different method or methods may be called by a command depending on how the command’s other code executes. Similarly, multiple commands can call the same method. We also list the methods’ argument signatures, which are used by their associated hooks.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Aurora Client API Method	Client API Method Argument Signature	Aurora Command Line Command
`create_job`	`self`, `config`	`job create`, `runtask`
`restart`	`self`, `job_key`, `shards`, `update_config`, `health_check_interval_seconds`	`job restart`
`kill_job`	`self`, `job_key`, `shards=None`	`job kill`
`start_cronjob`	`self`, `job_key`	`cron start`
`start_job_update`	`self`, `config`, `instances=None`	`update start`

+ +

Some specific examples:

+ +

pre_create_job executes when a create_job method is called, and before the create_job method itself executes.
post_cancel_update executes after a cancel_update method has successfully finished running.
err_kill_job executes when the kill_job method is called, but doesn’t successfully finish running.

+ +

Activating and Using Hooks

+ +

By default, hooks are inactive. If you do not want to use hooks, you do not need to make any changes to your code. If you do want to use hooks, you will need to alter your .aurora config file to activate them both for the configuration as a whole as well as for individual Jobs. And, of course, you will need to define in your config file what happens when a particular hook executes.

+ +

.aurora Config File Settings

+ +

You can define a top-level hooks variable in any .aurora config file. hooks is a list of all objects that define hooks used by Jobs defined in that config file. If you do not want to define any hooks for a configuration, hooks is optional.

hooks = [Object_with_defined_hooks1, Object_with_defined_hooks2]
+

+ +

Be careful when assembling a config file using include on multiple smaller config files. If there are multiple files that assign a value to hooks, only the last assignment made will stick. For example, if x.aurora has hooks = [a, b, c] and y.aurora has hooks = [d, e, f] and z.aurora has, in this order, include x.aurora and include y.aurora, the hooks value will be [d, e, f].

+ +

Also, for any Job that you want to use hooks with, its Job definition in the .aurora config file must set an enable_hooks flag to True (it defaults to False). By default, hooks are disabled and you must enable them for Jobs of your choice.

+ +

To summarize, to use hooks for a particular job, you must both activate hooks for your config file as a whole, and for that job. Activating hooks only for individual jobs won’t work, nor will only activating hooks for your config file as a whole. You must also specify the hooks’ defining object in the hooks variable.

+ +

Recall that .aurora config files are written in Pystachio. So the following turns on hooks for production jobs at cluster1 and cluster2, but leaves them off for similar jobs with a defined user role. Of course, you also need to list the objects that define the hooks in your config file’s hooks variable.

jobs = [
+        Job(enable_hooks = True, cluster = c, env = 'prod') for c in ('cluster1', 'cluster2')
+       ]
+jobs.extend(
+   Job(cluster = c, env = 'prod', role = getpass.getuser()) for c in ('cluster1', 'cluster2'))
+   # Hooks disabled for these jobs
+

+ +

Command Line

+ +

All Aurora Command Line commands now accept an .aurora config file as an optional parameter (some, of course, accept it as a required parameter). Whenever a command has a .aurora file parameter, any hooks specified and activated in the .aurora file can be used. For example:

aurora job restart cluster1/role/env/app myapp.aurora
+

+ +

The command activates any hooks specified and activated in myapp.aurora. For the restart command, that is the only thing the myapp.aurora parameter does. So, if the command was the following, since there is no .aurora config file to specify any hooks, no hooks on the restart command can run.

aurora job restart cluster1/role/env/app
+

+ +

Hooks Protocol

+ +

Any object defined in the .aurora config file can define hook methods. You should define your hook methods within a class, and then use the class name as a value in the hooks list in your config file.

+ +

Note that you can define other methods in the class that its hook methods can call; all the logic of a hook does not have to be in its definition.

+ +

The following example defines a class containing a pre_kill_job hook definition that calls another method defined in the class.

# Defines a method pre_kill_job
+class KillConfirmer(object):
+  def confirm(self, msg):
+    return raw_input(msg).lower() == 'yes'
+
+  def pre_kill_job(self, job_key, shards=None):
+    shards = ('shards %s' % shards) if shards is not None else 'all shards'
+    return self.confirm('Are you sure you want to kill %s (%s)? (yes/no): '
+                        % (job_key, shards))
+

+ +

pre_ Methods

+ +

pre_ methods have the signature:

pre_<API method name>(self, <associated method's signature>)
+

+ +

pre_ methods have the same signature as their associated method, with the addition of self as the first parameter. See the chart above for the mapping of parameters to methods. When writing pre_ methods, you can use the * and ** syntax to designate that all unspecified parameters are passed in a list to the *ed variable and all named parameters with values are passed as name/value pairs to the **ed variable.

+ +

If this method returns False, the API command call aborts.

+ +

err_ Methods

+ +

err_ methods have the signature:

err_<API method name>(self, exc, <associated method's signature>)
+

+ +

err_ methods have the same signature as their associated method, with the addition of a first parameter self and a second parameter exc. exc is either a result with responseCode other than ResponseCode.OK or an Exception. See the chart above for the mapping of parameters to methods. When writing err_ methods, you can use the * and ** syntax to designate that all unspecified parameters are passed in a list to the *ed variable and all named parameters with values are passed as name/value pairs to the **ed variable.

+ +

err_ method return codes are ignored.

+ +

post_ Methods

+ +

post_ methods have the signature:

post_<API method name>(self, result, <associated method signature>)
+

+ +

post_ method parameters are self, then result, followed by the same parameter signature as their associated method. result is the result of the associated method call. See the chart above for the mapping of parameters to methods. When writing post_ methods, you can use the * and ** syntax to designate that all unspecified arguments are passed in a list to the *ed parameter and all unspecified named arguments with values are passed as name/value pairs to the **ed parameter.

+ +

post_ method return codes are ignored.

+ +

Generic Hooks

+ +

There are seven Aurora API Methods which any of the three hook types can attach to. Thus, there are 21 possible hook/method combinations for a single .aurora config file. Say that you define pre_ and post_ hooks for the restart method. That leaves 19 undefined hook/method combinations; err_restart and the 3 pre_, post_, and err_ hooks for each of the other 6 hookable methods. You can define what happens when any of these otherwise undefined 19 hooks execute via a generic hook, whose signature is:

generic_hook(self, hook_config, event, method_name, result_or_err, args*, kw**)
+

+ +

where:

+ +

hook_config is a named tuple of config (the Pystashio config object) and job_key.
event is one of pre, err, or post, indicating which type of hook the genetic hook is standing in for. For example, assume no specific hooks were defined for the restart API command. If generic_hook is defined and activated, and restart is called, generic_hook will effectively run as pre_restart, post_restart, and err_restart. You can use a selection statement on this value so that generic_hook will act differently based on whether it is standing in for a pre_, post_, or err_ hook.
method_name is the Client API method name whose execution is causing this execution of the generic_hook.
args*, kw** are the API method arguments and keyword arguments respectively.
result_or_err is a tri-state parameter taking one of these three values:
+ +
1. None for pre_hooks
2. result for post_ nooks
3. exc for err_ hooks

+ +

Example:

# Overrides the standard do-nothing generic_hook by adding a log writing operation.
+from twitter.common import log
+  class Logger(object):
+    '''Adds to the log every time a hookable API method is called'''
+    def generic_hook(self, hook_config, event, method_name, result_or_err, *args, **kw)
+      log.info('%s: %s_%s of %s'
+               % (self.__class__.__name__, event, method_name, hook_config.job_key))
+

+ +

Hooks Process Checklist

+ +

In your .aurora config file, add a hooks variable. Note that you may want to define a .aurora file only for hook definitions and then include this file in multiple other config files that you want to use the same hooks.
+ +
hooks = []
In the hooks variable, list all objects that define hooks used by Jobs defined in this config:
+ +
hooks = [Objecthookdefiner1, Objecthookdefiner2]
For each job that uses hooks in this config file, add enable_hooks = True to the Job definition. Note that this is necessary even if you only want to use the generic hook.
Write your pre_, post_, and err_ hook definitions as part of an object definition in your .aurora config file.
If desired, write your generic_hook definition as part of an object definition in your .aurora config file. Remember, the object must be listed as a member of hooks.
If your Aurora command line command does not otherwise take an .aurora config file argument, add the appropriate .aurora file as an argument in order to define and activate the configuration’s hooks.

+ +

Quick Links

The ASF

+ + + Propchange: aurora/site/publish/documentation/0.14.0/reference/client-hooks/index.html ------------------------------------------------------------------------------ svn:eol-style = native Propchange: aurora/site/publish/documentation/0.14.0/reference/client-hooks/index.html ------------------------------------------------------------------------------ svn:keywords = Id Propchange: aurora/site/publish/documentation/0.14.0/reference/client-hooks/index.html ------------------------------------------------------------------------------ svn:mime-type = text/html Added: aurora/site/publish/documentation/0.14.0/reference/configuration-best-practices/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/reference/configuration-best-practices/index.html?rev=1748470&view=auto ============================================================================== --- aurora/site/publish/documentation/0.14.0/reference/configuration-best-practices/index.html (added) +++ aurora/site/publish/documentation/0.14.0/reference/configuration-best-practices/index.html Tue Jun 14 21:35:25 2016 @@ -0,0 +1,297 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Aurora Configuration Best Practices

+ +

Use As Few .aurora Files As Possible

+ +

When creating your .aurora configuration, try to keep all versions of +a particular job within the same .aurora file. For example, if you +have separate jobs for cluster1, cluster1 staging, cluster1 +testing, andcluster2, keep them as close together as possible.

+ +

Constructs shared across multiple jobs owned by your team (e.g. +team-level defaults or structural templates) can be split into separate +.aurorafiles and included via the include directive.

+ +

Avoid Boilerplate

+ +

If you see repetition or find yourself copy and pasting any parts of +your configuration, it’s likely an opportunity for templating. Take the +example below:

+ +

redundant.aurora contains:

download = Process(
+  name = 'download',
+  cmdline = 'wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2',
+  max_failures = 5,
+  min_duration = 1)
+
+unpack = Process(
+  name = 'unpack',
+  cmdline = 'rm -rf Python-2.7.3 && tar xzf Python-2.7.3.tar.bz2',
+  max_failures = 5,
+  min_duration = 1)
+
+build = Process(
+  name = 'build',
+  cmdline = 'pushd Python-2.7.3 && ./configure && make && popd',
+  max_failures = 1)
+
+email = Process(
+  name = 'email',
+  cmdline = 'echo Success | mail feynman@tmc.com',
+  max_failures = 5,
+  min_duration = 1)
+
+build_python = Task(
+  name = 'build_python',
+  processes = [download, unpack, build, email],
+  constraints = [Constraint(order = ['download', 'unpack', 'build', 'email'])])
+

+ +

As you’ll notice, there’s a lot of repetition in the Process +definitions. For example, almost every process sets a max_failures +limit to 5 and a min_duration to 1. This is an opportunity for factoring +into a common process template.

+ +

Furthermore, the Python version is repeated everywhere. This can be +bound via structural templating as described in the Advanced Binding +section.

+ +

less_redundant.aurora contains:

class Python(Struct):
+  version = Required(String)
+  base = Default(String, 'Python-{{version}}')
+  package = Default(String, '{{base}}.tar.bz2')
+
+ReliableProcess = Process(
+  max_failures = 5,
+  min_duration = 1)
+
+download = ReliableProcess(
+  name = 'download',
+  cmdline = 'wget http://www.python.org/ftp/python/{{python.version}}/{{python.package}}')
+
+unpack = ReliableProcess(
+  name = 'unpack',
+  cmdline = 'rm -rf {{python.base}} && tar xzf {{python.package}}')
+
+build = ReliableProcess(
+  name = 'build',
+  cmdline = 'pushd {{python.base}} && ./configure && make && popd',
+  max_failures = 1)
+
+email = ReliableProcess(
+  name = 'email',
+  cmdline = 'echo Success | mail {{role}}@foocorp.com')
+
+build_python = SequentialTask(
+  name = 'build_python',
+  processes = [download, unpack, build, email]).bind(python = Python(version = "2.7.3"))
+

+ +

Thermos Uses bash, But Thermos Is Not bash

+ +

Bad

+ +

Many tiny Processes makes for harder to manage configurations.

copy = Process(
+  name = 'copy',
+  cmdline = 'rcp user@my_machine:my_application .'
+ )
+
+ unpack = Process(
+   name = 'unpack',
+   cmdline = 'unzip app.zip'
+ )
+
+ remove = Process(
+   name = 'remove',
+   cmdline = 'rm -f app.zip'
+ )
+
+ run = Process(
+   name = 'app',
+   cmdline = 'java -jar app.jar'
+ )
+
+ run_task = Task(
+   processes = [copy, unpack, remove, run],
+   constraints = order(copy, unpack, remove, run)
+ )
+

+ +

Good

+ +

Each cmdline runs in a bash subshell, so you have the full power of +bash. Chaining commands with && or || is almost always the right +thing to do.

+ +

Also for Tasks that are simply a list of processes that run one after +another, consider using the SequentialTask helper which applies a +linear ordering constraint for you.

stage = Process(
+  name = 'stage',
+  cmdline = 'rcp user@my_machine:my_application . && unzip app.zip && rm -f app.zip')
+
+run = Process(name = 'app', cmdline = 'java -jar app.jar')
+
+run_task = SequentialTask(processes = [stage, run])
+

+ +

Rarely Use Functions In Your Configurations

+ +

90% of the time you define a function in a .aurora file, you’re +probably Doing It Wrong™.

+ +

Bad

def get_my_task(name, user, cpu, ram, disk):
+  return Task(
+    name = name,
+    user = user,
+    processes = [STAGE_PROCESS, RUN_PROCESS],
+    constraints = order(STAGE_PROCESS, RUN_PROCESS),
+    resources = Resources(cpu = cpu, ram = ram, disk = disk)
+ )
+
+ task_one = get_my_task('task_one', 'feynman', 1.0, 32*MB, 1*GB)
+ task_two = get_my_task('task_two', 'feynman', 2.0, 64*MB, 1*GB)
+

+ +

Good

+ +

This one is more idiomatic. Forced keyword arguments prevents accidents, +e.g. constructing a task with “32*MB” when you mean 32MB of ram and not +disk. Less proliferation of task-construction techniques means +easier-to-read, quicker-to-understand, and a more composable +configuration.

TASK_TEMPLATE = SequentialTask(
+  user = 'wickman',
+  processes = [STAGE_PROCESS, RUN_PROCESS],
+)
+
+task_one = TASK_TEMPLATE(
+  name = 'task_one',
+  resources = Resources(cpu = 1.0, ram = 32*MB, disk = 1*GB) )
+
+task_two = TASK_TEMPLATE(
+  name = 'task_two',
+  resources = Resources(cpu = 2.0, ram = 64*MB, disk = 1*GB)
+)
+

+ +

Quick Links

The ASF

+ + + Propchange: aurora/site/publish/documentation/0.14.0/reference/configuration-best-practices/index.html ------------------------------------------------------------------------------ svn:eol-style = native Propchange: aurora/site/publish/documentation/0.14.0/reference/configuration-best-practices/index.html ------------------------------------------------------------------------------ svn:keywords = Id Propchange: aurora/site/publish/documentation/0.14.0/reference/configuration-best-practices/index.html ------------------------------------------------------------------------------ svn:mime-type = text/html

Documentation + + + 0.14.0 + (latest) + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Aurora Scheduler Storage

Overview

Storage Semantics

Reads, writes, modifications

Read lifecycle

Write lifecycle

Atomicity, consistency and isolation

Population on restart

Quick Links

The ASF

Documentation + + + 0.14.0 + (latest) + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Client Cluster Configuration

Details

name

slave_root

slave_run_directory

zk

zk_port

scheduler_zk_path

scheduler_uri

proxy_url

auth_mechanism

Quick Links

The ASF

Documentation + + + 0.14.0 + (latest) + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Aurora Client Commands

Introduction

Cluster Configuration

Job Keys

Modifying Aurora Client Commands

Regular Jobs

Creating and Running a Job

Running a Command On a Running Job

Killing a Job

Adding Instances

Updating a Job

Renaming a Job

Restarting Jobs

Cron Jobs

Comparing Jobs

Viewing/Examining Jobs

Listing Jobs

Inspecting a Job

Checking Your Quota

Finding a Job on Web UI

Getting Job Status

Opening the Web UI

SSHing to a Specific Task Machine

Templating Command Arguments

Quick Links

The ASF

Documentation + + + 0.14.0 + (latest) + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Hooks for Aurora Client API

Hook Types

Execution Order

Hookable Methods

Activating and Using Hooks

.aurora Config File Settings

Command Line

Hooks Protocol

pre_ Methods

err_ Methods

post_ Methods

Generic Hooks

Hooks Process Checklist

Quick Links

The ASF

Documentation + + + 0.14.0 + (latest) + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Aurora Configuration Best Practices

Use As Few .aurora Files As Possible

Avoid Boilerplate

Thermos Uses bash, But Thermos Is Not bash

Bad

Good

Rarely Use Functions In Your Configurations

Bad

Good

Quick Links

The ASF

Documentation + +

Documentation + +

`name`

`slave_root`

`slave_run_directory`

`zk`

`zk_port`

`scheduler_zk_path`

`scheduler_uri`

`proxy_url`

`auth_mechanism`

Documentation + +

Documentation + +

Documentation + +