Return-Path: X-Original-To: apmail-aurora-commits-archive@minotaur.apache.org Delivered-To: apmail-aurora-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2DD8A10BC8 for ; Thu, 27 Feb 2014 04:08:07 +0000 (UTC) Received: (qmail 13003 invoked by uid 500); 27 Feb 2014 04:08:06 -0000 Delivered-To: apmail-aurora-commits-archive@aurora.apache.org Received: (qmail 12956 invoked by uid 500); 27 Feb 2014 04:08:01 -0000 Mailing-List: contact commits-help@aurora.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@aurora.incubator.apache.org Delivered-To: mailing list commits@aurora.incubator.apache.org Received: (qmail 12949 invoked by uid 99); 27 Feb 2014 04:07:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Feb 2014 04:07:59 +0000 X-ASF-Spam-Status: No, hits=-2000.5 required=5.0 tests=ALL_TRUSTED,NORMAL_HTTP_TO_IP,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 27 Feb 2014 04:07:54 +0000 Received: (qmail 12887 invoked by uid 99); 27 Feb 2014 04:07:32 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Feb 2014 04:07:32 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 10A7992E320; Thu, 27 Feb 2014 04:07:31 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: wfarner@apache.org To: commits@aurora.incubator.apache.org Message-Id: <3496d584215b419f894bd6dca189284b@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: git commit: Make markdown TOC and anchors more plaintext-friendly. Date: Thu, 27 Feb 2014 04:07:31 +0000 (UTC) X-Virus-Checked: Checked by ClamAV on apache.org Repository: incubator-aurora Updated Branches: refs/heads/master 105b600ec -> ccc6694f3 Make markdown TOC and anchors more plaintext-friendly. Reviewed at https://reviews.apache.org/r/18528/ Project: http://git-wip-us.apache.org/repos/asf/incubator-aurora/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-aurora/commit/ccc6694f Tree: http://git-wip-us.apache.org/repos/asf/incubator-aurora/tree/ccc6694f Diff: http://git-wip-us.apache.org/repos/asf/incubator-aurora/diff/ccc6694f Branch: refs/heads/master Commit: ccc6694f38d1015ed0debbd66726c2e4a9033b14 Parents: 105b600 Author: Bill Farner Authored: Wed Feb 26 20:07:18 2014 -0800 Committer: Bill Farner Committed: Wed Feb 26 20:07:18 2014 -0800 ---------------------------------------------------------------------- docs/configurationreference.md | 257 +++++++++++++++++++----------------- 1 file changed, 137 insertions(+), 120 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/ccc6694f/docs/configurationreference.md ---------------------------------------------------------------------- diff --git a/docs/configurationreference.md b/docs/configurationreference.md index 3c76d59..9d5c340 100644 --- a/docs/configurationreference.md +++ b/docs/configurationreference.md @@ -1,29 +1,44 @@ Aurora + Thermos Configuration Reference ======================================== - -[Introduction](#Introduction) -[Process Schema](#Process Schema) -    [`Process` Objects](#ProcessObject) -[Task Schema](#TaskSchema) -    [`Task` Object](#TaskObject) -    [`Constraint` Object](#ConstraintObject) -    [`Resource` Object](#ResourceObject) -[Job Schema](#JobSchema) -    [`Job` Objects](#JobObject) -    [Services](#Services) -    [`UpdateConfig` Objects](#UpdateConfigObjects) -    [`HealthCheckConfig` Objects](#HealthCheckConfigObject) -[Specifying Scheduling Constraints](#SchedulingConstraints) -[Template Namespaces](#TemplateNamespaces) -    [`mesos` Namespace](#mesosNamespace) -    [`thermos` Namespace](#thermosNamespace) -[Basic Examples](#BasicExamples) -    [`hello_world.aurora`](#hello_world.aurora) -    [Environment Tailoring](#EnvironmentTailoring) - -Introduction -======================================= +- [Aurora + Thermos Configuration Reference](#aurora--thermos-configuration-reference) +- [Introduction](#introduction) +- [Process Schema](#process-schema) + - [Process Objects](#process-objects) + - [name](#name) + - [cmdline](#cmdline) + - [max_failures](#max_failures) + - [daemon](#daemon) + - [ephemeral](#ephemeral) + - [min_duration](#min_duration) + - [final](#final) +- [Task Schema](#task-schema) + - [Task Object](#task-object) + - [name](#name-1) + - [processes](#processes) + - [constraints](#constraints) + - [resources](#resources) + - [max_failures](#max_failures-1) + - [max_concurrency](#max_concurrency) + - [finalization_wait](#finalization_wait) + - [Constraint Object](#constraint-object) + - [Resource Object](#resource-object) +- [Job Schema](#job-schema) + - [Job Objects](#job-objects) + - [Services](#services) + - [UpdateConfig Objects](#updateconfig-objects) + - [HealthCheckConfig Objects](#healthcheckconfig-objects) +- [Specifying Scheduling Constraints](#specifying-scheduling-constraints) +- [Template Namespaces](#template-namespaces) + - [mesos Namespace](#mesos-namespace) + - [thermos Namespace](#thermos-namespace) +- [Basic Examples](#basic-examples) + - [hello_world.aurora](#hello_worldaurora) + - [Environment Tailoring](#environment-tailoring) + - [hello_world_productionized.aurora](#hello_world_productionizedaurora) + +Introduction +============ Don't know where to start? The Aurora configuration schema is very powerful, and configurations can become quite complex for advanced use @@ -31,18 +46,18 @@ cases. For examples of simple configurations to get something up and running quickly, check out the [Tutorial](tutorial.md). When you feel comfortable with the basics, move -on to the [Configuration Tutorial](configurationtutorial.md) for more in-depth coverage of configuration -design. +on to the [Configuration Tutorial](configurationtutorial.md) for more in-depth coverage of +configuration design. For additional basic configuration examples, see [the end of this document](#BasicExamples). -Process Schema -=========================================== +Process Schema +============== Process objects consist of required `name` and `cmdline` attributes. You can customize Process behavior with its optional attributes. Remember, Processes are handled by Thermos. -### `Process` Objects +### Process Objects @@ -88,19 +103,19 @@ behavior with its optional attributes. Remember, Processes are handled by Thermo
-#### `name` +#### name The name is any valid UNIX filename string (specifically no slashes, NULLs or leading periods). Within a Task object, each Process name must be unique. -#### `cmdline` +#### cmdline The command line run by the process. The command line is invoked in a bash subshell, so can involve fully-blown bash scripts. However, nothing is supplied for command-line arguments so `$*` is unspecified. -#### `max_failures` +#### max_failures The maximum number of failures (non-zero exit statuses) this process can have before being marked permanently failed and not retried. If a @@ -113,7 +128,7 @@ indefinitely until it achieves a successful (zero) exit status. It retries at most once every `min_duration` seconds to prevent an effective denial of service attack on the coordinating Thermos scheduler. -#### `daemon` +#### daemon By default, Thermos processes are non-daemon. If `daemon` is set to True, a successful (zero) exit status does not prevent future process runs. @@ -125,7 +140,7 @@ for very short-lived processes because of the accumulation of checkpointed state for each process run. When running in Mesos specifically, `max_failures` is capped at 100. -#### `ephemeral` +#### ephemeral By default, Thermos processes are non-ephemeral. If `ephemeral` is set to True, the process' status is not used to determine if its containing task @@ -135,14 +150,14 @@ that periodically checkpoints its log files to a centralized data store. The task is considered finished once the webserver process has completed, regardless of the logsaver's current status. -#### `min_duration` +#### min_duration Processes may succeed or fail multiple times during a single task's duration. Each of these is called a *process run*. `min_duration` is the minimum number of seconds the scheduler waits before running the same process. -#### `final` +#### final Processes can be grouped into two classes: ordinary processes and finalizing processes. By default, Thermos processes are ordinary. They @@ -159,8 +174,8 @@ vice-versa, however finalizing processes may depend upon other finalizing processes and otherwise run as a typical process schedule. -Task Schema -==================================== +Task Schema +=========== Tasks fundamentally consist of a `name` and a list of Process objects stored as the value of the `processes` attribute. Processes can be further constrained with @@ -168,7 +183,7 @@ value of the `processes` attribute. Processes can be further constrained with `processes` list, so for simple `Task` objects with one Process, `name` can be omitted. In Mesos, `resources` is also required. -### `Task` Object +### Task Object @@ -214,16 +229,17 @@ can be omitted. In Mesos, `resources` is also required.
-`name` +#### name -`name` is a string denoting the name of this task. It defaults to the name of the first Process in the list of Processes associated with the `processes` attribute. +`name` is a string denoting the name of this task. It defaults to the name of the first Process in +the list of Processes associated with the `processes` attribute. -`processes` +#### processes `processes` is an unordered list of `Process` objects. To constrain the order in which they run, use `constraints`. -`constraints` +##### constraints A list of `Constraint` objects. Currently it supports only one type, the `order` constraint. `order` is a list of process names @@ -247,11 +263,12 @@ is shorthand for [Constraint(order = [process1.name(), process2.name()])] -`resources` +#### resources -Takes a `Resource` object, which specifies the amounts of CPU, memory, and disk space resources to allocate to the Task. +Takes a `Resource` object, which specifies the amounts of CPU, memory, and disk space resources +to allocate to the Task. -`max_failures` +#### max_failures `max_failures` is the number of times processes that are part of this Task can fail before the entire Task is marked for failure. @@ -273,7 +290,7 @@ first run. The task would succeed despite only allowing for two failed processes. To be more specific, there would be 10 failed process runs yet 1 failed process. -`max_concurrency` +#### max_concurrency For Tasks with a number of expensive but otherwise independent processes, you may want to limit the amount of concurrency @@ -287,26 +304,26 @@ For example, the following task spawns 180 Processes ("mappers") to compute individual elements of a 180 degree sine table, all dependent upon one final Process ("reducer") to tabulate the results: - def make_mapper(id): - return Process( - name = "mapper%03d" % id, - cmdline = "echo 'scale=50;s(%d\*4\*a(1)/180)' | bc -l > - temp.sine_table.%03d" % (id, id)) + def make_mapper(id): + return Process( + name = "mapper%03d" % id, + cmdline = "echo 'scale=50;s(%d\*4\*a(1)/180)' | bc -l > + temp.sine_table.%03d" % (id, id)) - def make_reducer(): - return Process(name = "reducer", cmdline = "cat temp.\* | nl \> sine\_table.txt - && rm -f temp.\*") + def make_reducer(): + return Process(name = "reducer", cmdline = "cat temp.\* | nl \> sine\_table.txt + && rm -f temp.\*") - processes = map(make_mapper, range(180)) + processes = map(make_mapper, range(180)) - task = Task( - name = "mapreduce", - processes = processes + [make\_reducer()], - constraints = [Constraint(order = [mapper.name(), 'reducer']) for mapper - in processes], - max_concurrency = 8) + task = Task( + name = "mapreduce", + processes = processes + [make\_reducer()], + constraints = [Constraint(order = [mapper.name(), 'reducer']) for mapper + in processes], + max_concurrency = 8) -`finalization_wait` +#### finalization_wait Tasks have three active stages: `ACTIVE`, `CLEANING`, and `FINALIZING`. The `ACTIVE` stage is when ordinary processes run. This stage lasts as @@ -327,7 +344,7 @@ Client applications with higher priority may force a shorter finalization wait (e.g. through parameters to `thermos kill`), so this is mostly a best-effort signal. -### `Constraint` Object +### Constraint Object Current constraint objects only support a single ordering constraint, `order`, which specifies its processes run sequentially in the order given. By @@ -349,9 +366,11 @@ ordering constraints. -### `Resource` Object +### Resource Object -Specifies the amount of CPU, Ram, and disk resources the task needs. See the [Resource Isolation document](resourceisolation.md) for suggested values and to understand how resources are allocated. +Specifies the amount of CPU, Ram, and disk resources the task needs. See the +[Resource Isolation document](resourceisolation.md) for suggested values and to understand how +resources are allocated. @@ -378,10 +397,10 @@ Specifies the amount of CPU, Ram, and disk resources the task needs. See the [Re
-Job Schema -================================== +Job Schema +========== -### `Job` Objects +### Job Objects @@ -473,7 +492,7 @@ Specifies the amount of CPU, Ram, and disk resources the task needs. See the [Re
-### Services +### Services Jobs with the `service` flag set to True are called Services. The `Service` alias can be used as shorthand for `Job` with `service=True`. @@ -483,7 +502,7 @@ Jobs without the service bit set only restart up to `max_task_failures` times and only if they terminated unsuccessfully either due to human error or machine failure. -### `UpdateConfig` Objects +### UpdateConfig Objects Parameters for controlling the rate and policy of rolling updates. @@ -517,7 +536,7 @@ Parameters for controlling the rate and policy of rolling updates. -### `HealthCheckConfig` Objects +### HealthCheckConfig Objects Parameters for controlling a task's health checks via HTTP. @@ -546,8 +565,8 @@ Parameters for controlling a task's health checks via HTTP. -Specifying Scheduling Constraints -===================================================================== +Specifying Scheduling Constraints +================================= Most users will not need to specify constraints explicitly, as the scheduler automatically inserts reasonable defaults that attempt to @@ -586,28 +605,28 @@ You can also control machine diversity using constraints. The below constraint ensures that no more than two instances of your job may run on a single host. Think of this as a "group by" limit. - constraints = { - 'host': 'limit:2', - } + constraints = { + 'host': 'limit:2', + } Likewise, you can use constraints to control rack diversity, e.g. at most one task per rack: - constraints = { - 'rack': 'limit:1', - } + constraints = { + 'rack': 'limit:1', + } Use these constraints sparingly as they can dramatically reduce Tasks' schedulability. -Template Namespaces -==================================================== +Template Namespaces +=================== Currently, a few Pystachio namespaces have special semantics. Using them in your configuration allow you to tailor application behavior through environment introspection or interact in special ways with the Aurora client or Aurora-provided services. -### `mesos` Namespace +### mesos Namespace The `mesos` namespace contains the `instance` variable that can be used to distinguish between Task replicas. @@ -622,7 +641,7 @@ to distinguish between Task replicas. -### `thermos` Namespace +### thermos Namespace The `thermos` namespace contains variables that work directly on the Thermos platform in addition to Aurora. This namespace is fully @@ -652,67 +671,65 @@ configuration, it is automatically extracted and auto-populated by Aurora, but must be specified with, for example, `thermos -P http:12345` to map `http` to port 12345 when running via the CLI. -Basic Examples -========================================== +Basic Examples +============== These are provided to give a basic understanding of simple Aurora jobs. -### `hello_world.aurora` +### hello_world.aurora Put the following in a file named `hello_world.aurora`, substituting your own values for values such as `cluster`s. - import os - hello_world_process = Process(name = 'hello_world', cmdline = 'echo hello world') + import os + hello_world_process = Process(name = 'hello_world', cmdline = 'echo hello world') - hello_world_task = Task( - resources = Resources(cpu = 0.1, ram = 16 * MB, disk = 16 * MB), - processes = [hello_world_process]) + hello_world_task = Task( + resources = Resources(cpu = 0.1, ram = 16 * MB, disk = 16 * MB), + processes = [hello_world_process]) - hello_world_job = Job( - cluster = 'cluster1', - role = os.getenv('USER'), - task = hello_world_task) + hello_world_job = Job( + cluster = 'cluster1', + role = os.getenv('USER'), + task = hello_world_task) - jobs = [hello_world_job] + jobs = [hello_world_job] Then issue the following commands to create and kill the job, using your own values for the job key. - `aurora create cluster1/$USER/test/hello_world hello_world.aurora` + aurora create cluster1/$USER/test/hello_world hello_world.aurora - `aurora kill cluster1/$USER/test/hello_world ` + aurora kill cluster1/$USER/test/hello_world -### Environment Tailoring +### Environment Tailoring -### `hello_world_productionized.aurora` +#### hello_world_productionized.aurora Put the following in a file named `hello_world_productionized.aurora`, substituting your own values for values such as `cluster`s. - include('hello_world.aurora') + include('hello_world.aurora') - production_resources = Resources(cpu = 1.0, ram = 512 * MB, disk = 2 * GB) - staging_resources = Resources(cpu = 0.1, ram = 32 * MB, disk = 512 * MB) - hello_world_template = hello_world( - name = "hello_world-{{cluster}}" - task = hello_world(resources=production_resources)) + production_resources = Resources(cpu = 1.0, ram = 512 * MB, disk = 2 * GB) + staging_resources = Resources(cpu = 0.1, ram = 32 * MB, disk = 512 * MB) + hello_world_template = hello_world( + name = "hello_world-{{cluster}}" + task = hello_world(resources=production_resources)) - jobs = [ - # production jobs - hello_world_template(cluster = 'cluster1', instances = 25), - hello_world_template(cluster = 'cluster2', instances = 15), + jobs = [ + # production jobs + hello_world_template(cluster = 'cluster1', instances = 25), + hello_world_template(cluster = 'cluster2', instances = 15), - # staging jobs - hello_world_template( - cluster = 'local', - instances = 1, - task = hello_world(resources=staging_resources)), - ] + # staging jobs + hello_world_template( + cluster = 'local', + instances = 1, + task = hello_world(resources=staging_resources)), + ] Then issue the following commands to create and kill the job, using your own values for the job key - aurora create cluster1/$USER/test/hello_world-cluster1 hello_world_productionized.aurora - - aurora kill cluster1/$USER/test/hello_world-cluster1 - + aurora create cluster1/$USER/test/hello_world-cluster1 hello_world_productionized.aurora + aurora kill cluster1/$USER/test/hello_world-cluster1