; Tue, 21 Feb 2017 20:55:11 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1783940 [5/20] - in /aurora/site: data/ publish/ publish/blog/ publish/blog/aurora-0-17-0-released/ publish/documentation/0.10.0/ publish/documentation/0.10.0/build-system/ publish/documentation/0.10.0/client-cluster-configuration/ publish... Date: Tue, 21 Feb 2017 20:55:06 -0000 To: commits@aurora.apache.org From: serb@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20170221205511.4CE623A3BED@svn01-us-west.apache.org> archived-at: Tue, 21 Feb 2017 20:55:16 -0000 Added: aurora/site/publish/documentation/0.17.0/development/design-documents/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/development/design-documents/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/development/design-documents/index.html (added) +++ aurora/site/publish/documentation/0.17.0/development/design-documents/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,153 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Design Documents

+ +

Since its inception as an Apache project, larger feature additions to the +Aurora code base are discussed in form of design documents. Design documents +are living documents until a consensus has been reached to implement a feature +in the proposed form.

+ +

Current and past documents:

+ +

Design documents can be found in the Aurora issue tracker via the query project = AURORA AND text ~ "docs.google.com" ORDER BY created.

+ +

Quick Links

The ASF

© 2014-2017 Apache Software Foundation. Licensed under the Apache License v2.0. The Aurora Borealis IX photo displayed on the homepage is available under a Creative Commons BY-NC-ND 2.0 license. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.

+ + + Added: aurora/site/publish/documentation/0.17.0/development/design/command-hooks/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/development/design/command-hooks/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/development/design/command-hooks/index.html (added) +++ aurora/site/publish/documentation/0.17.0/development/design/command-hooks/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,233 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Command Hooks for the Aurora Client

+ +

Introduction/Motivation

+ +

We’ve got hooks in the client that surround API calls. These are +pretty awkward, because they don’t correlate with user actions. For +example, suppose we wanted a policy that said users weren’t allowed to +kill all instances of a production job at once.

+ +

Right now, all that we could hook would be the “killJob” api call. But +kill (at least in newer versions of the client) normally runs in +batches. If a user called killall, what we would see on the API level +is a series of “killJob” calls, each of which specified a batch of +instances. We woudn’t be able to distinguish between really killing +all instances of a job (which is forbidden under this policy), and +carefully killing in batches (which is permitted.) In each case, the +hook would just see a series of API calls, and couldn’t find out what +the actual command being executed was!

+ +

For most policy enforcement, what we really want to be able to do is +look at and vet the commands that a user is performing, not the API +calls that the client uses to implement those commands.

+ +

So I propose that we add a new kind of hooks, which surround noun/verb +commands. A hook will register itself to handle a collection of (noun, +verb) pairs. Whenever any of those noun/verb commands are invoked, the +hooks methods will be called around the execution of the verb. A +pre-hook will have the ability to reject a command, preventing the +verb from being executed.

+ +

Registering Hooks

+ +

These hooks will be registered via configuration plugins. A configuration plugin +can register hooks using an API. Hooks registered this way are, effectively, +hardwired into the client executable.

+ +

The order of execution of hooks is unspecified: they may be called in +any order. There is no way to guarantee that one hook will execute +before some other hook.

+ +

Global Hooks

+ +

Commands registered by the python call are called global hooks, +because they will run for all configurations, whether or not they +specify any hooks in the configuration file.

+ +

In the implementation, hooks are registered in the module +apache.aurora.client.cli.command_hooks, using the class +GlobalCommandHookRegistry. A global hook can be registered by calling +GlobalCommandHookRegistry.register_command_hook in a configuration plugin.

+ +

The API

class CommandHook(object)
+  @property
+  def name(self):
+    """Returns a name for the hook."
+
+  def get_nouns(self):
+    """Return the nouns that have verbs that should invoke this hook."""
+
+  def get_verbs(self, noun):
+    """Return the verbs for a particular noun that should invoke his hook."""
+
+  @abstractmethod
+  def pre_command(self, noun, verb, context, commandline):
+    """Execute a hook before invoking a verb.
+    * noun: the noun being invoked.
+    * verb: the verb being invoked.
+    * context: the context object that will be used to invoke the verb.
+      The options object will be initialized before calling the hook
+    * commandline: the original argv collection used to invoke the client.
+    Returns: True if the command should be allowed to proceed; False if the command
+    should be rejected.
+    """
+
+  def post_command(self, noun, verb, context, commandline, result):
+    """Execute a hook after invoking a verb.
+    * noun: the noun being invoked.
+    * verb: the verb being invoked.
+    * context: the context object that will be used to invoke the verb.
+      The options object will be initialized before calling the hook
+    * commandline: the original argv collection used to invoke the client.
+    * result: the result code returned by the verb.
+    Returns: nothing
+    """
+
+class GlobalCommandHookRegistry(object):
+  @classmethod
+  def register_command_hook(self, hook):
+    pass
+

+ +

Skipping Hooks

+ +

To skip a hook, a user uses a command-line option, --skip-hooks. The option can either +specify specific hooks to skip, or “all”:

+ +

aurora --skip-hooks=all job create east/bozo/devel/myjob will create a job +without running any hooks.
aurora --skip-hooks=test,iq create east/bozo/devel/myjob will create a job, +and will skip only the hooks named “test” and “iq”.

+ +

Quick Links

The ASF

+ + + Added: aurora/site/publish/documentation/0.17.0/development/scheduler/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/development/scheduler/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/development/scheduler/index.html (added) +++ aurora/site/publish/documentation/0.17.0/development/scheduler/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,238 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Developing the Aurora Scheduler

+ +

The Aurora scheduler is written in Java code and built with Gradle.

+ +

Prerequisite

+ +

When using Apache Aurora checked out from the source repository or the binary +distribution, the Gradle wrapper and JavaScript dependencies are provided. +However, you need to manually install them when using the source release +downloads:

+ +

Install Gradle following the instructions on the Gradle web site
From the root directory of the Apache Aurora project generate the Gradle +wrapper by running:
+ +
gradle wrapper

+ +

Getting Started

+ +

You will need Java 8 installed and on your PATH or unzipped somewhere with JAVA_HOME set. Then

./gradlew tasks
+

+ +

will bootstrap the build system and show available tasks. This can take a while the first time you +run it but subsequent runs will be much faster due to cached artifacts.

+ +

Running the Tests

+ +

Aurora has a comprehensive unit test suite. To run the tests use

./gradlew build
+

+ +

Gradle will only re-run tests when dependencies of them have changed. To force a re-run of all +tests use

./gradlew clean build
+

+ +

Running the build with code quality checks

+ +

To speed up development iteration, the plain gradle commands will not run static analysis tools. +However, you should run these before posting a review diff, and always run this before pushing a +commit to origin/master.

./gradlew build -Pq
+

+ +

Running integration tests

+ +

To run the same tests that are run in the Apache Aurora continuous integration +environment:

./build-support/jenkins/build.sh
+

+ +

In addition, there is an end-to-end test that runs a suite of aurora commands +using a virtual cluster:

./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
+

+ +

Creating a bundle for deployment

+ +

Gradle can create a zip file containing Aurora, all of its dependencies, and a launch script with

./gradlew distZip
+

+ +

or a tar file containing the same files with

./gradlew distTar
+

+ +

The output file will be written to dist/distributions/aurora-scheduler.zip or +dist/distributions/aurora-scheduler.tar.

+ +

Developing Aurora Java code

+ +

Setting up an IDE

+ +

Gradle can generate project files for your IDE. To generate an IntelliJ IDEA project run

./gradlew idea
+

+ +

and import the generated aurora.ipr file.

+ +

Adding or Upgrading a Dependency

+ +

New dependencies can be added from Maven central by adding a compile dependency to build.gradle. +For example, to add a dependency on com.example’s example-lib 1.0 add this block:

compile 'com.example:example-lib:1.0'
+

+ +

NOTE: Anyone thinking about adding a new dependency should first familiarize themselves with the +Apache Foundation’s third-party licensing +policy.

+ +

Developing the Aurora Build System

+ +

Bootstrapping Gradle

+ +

The following files were autogenerated by gradle wrapper using gradle’s +Wrapper plugin and +should not be modified directly:

./gradlew
+./gradlew.bat
+./gradle/wrapper/gradle-wrapper.jar
+./gradle/wrapper/gradle-wrapper.properties
+

+ +

To upgrade Gradle unpack the new version somewhere, run /path/to/new/gradle wrapper in the +repository root and commit the changed files.

+ +

Quick Links

The ASF

+ + + Added: aurora/site/publish/documentation/0.17.0/development/thermos/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/development/thermos/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/development/thermos/index.html (added) +++ aurora/site/publish/documentation/0.17.0/development/thermos/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,249 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

The Python components of Aurora are built using Pants.

+ +

Python Build Conventions

+ +

The Python code is laid out according to the following conventions:

+ +

1 BUILD per 3rd level directory. For a list of current top-level packages run:

% find src/main/python -maxdepth 3 -mindepth 3 -type d |\
+while read dname; do echo $dname |\
+    sed 's@src/main/python/\(.*\)/\(.*\)/\(.*\).*@\1.\2.\3@'; done
+

Each BUILD file exports 1 +python_library +that provides a +setup_py +containing each +python_binary +in the BUILD file, named the same as the directory it’s in so that it can be referenced +without a ’:’ character. The sources field in the python_library will almost always be +rglobs('*.py').
Other BUILD files may only depend on this single public python_library +target. Any other target is considered a private implementation detail and +should be prefixed with an _.
python_binary targets are always named the same as the exported console script.
python_binary targets must have identical dependencies to the python_library exported +by the package and must use entry_point.
+ +
The means a PEX file generated by pants will contain exactly the same files that will be +available on the PYTHONPATH in the case of pip install of the corresponding library +target. This will help our migration off of Pants in the future.

+ +

Annotated example - apache.thermos.runner

% find src/main/python/apache/thermos/runner
+src/main/python/apache/thermos/runner
+src/main/python/apache/thermos/runner/__init__.py
+src/main/python/apache/thermos/runner/thermos_runner.py
+src/main/python/apache/thermos/runner/BUILD
+% cat src/main/python/apache/thermos/runner/BUILD
+# License boilerplate omitted
+import os
+
+
+# Private target so that a setup_py can exist without a circular dependency. Only targets within
+# this file should depend on this.
+python_library(
+  name = '_runner',
+  # The target covers every python file under this directory and subdirectories.
+  sources = rglobs('*.py'),
+  dependencies = [
+    '3rdparty/python:twitter.common.app',
+    '3rdparty/python:twitter.common.log',
+    # Source dependencies are always referenced without a ':'.
+    'src/main/python/apache/thermos/common',
+    'src/main/python/apache/thermos/config',
+    'src/main/python/apache/thermos/core',
+  ],
+)
+
+# Binary target for thermos_runner.pex. Nothing should depend on this - it's only used as an
+# argument to ./pants binary.
+python_binary(
+  name = 'thermos_runner',
+  # Use entry_point, not source so the files used here are the same ones tests see.
+  entry_point = 'apache.thermos.bin.thermos_runner',
+  dependencies = [
+    # Notice that we depend only on the single private target from this BUILD file here.
+    ':_runner',
+  ],
+)
+
+# The public library that everyone importing the runner symbols uses.
+# The test targets and any other dependent source code should depend on this.
+python_library(
+  name = 'runner',
+  dependencies = [
+    # Again, notice that we depend only on the single private target from this BUILD file here.
+    ':_runner',
+  ],
+  # We always provide a setup_py. This will cause any dependee libraries to automatically
+  # reference this library in their requirements.txt rather than copy the source files into their
+  # sdist.
+  provides = setup_py(
+    # Conventionally named and versioned.
+    name = 'apache.thermos.runner',
+    version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(),
+  ).with_binaries({
+    # Every binary in this file should also be repeated here.
+    # Always use the dict-form of .with_binaries so that commands with dashes in their names are
+    # supported.
+    # The console script name is always the same as the PEX with .pex stripped.
+    'thermos_runner': ':thermos_runner',
+  }),
+)
+

+ +

Thermos Test resources

+ +

The Aurora source repository and distributions contain several +binary files to +qualify the backwards-compatibility of thermos with checkpoint data. Since +thermos persists state to disk, to be read by the thermos observer), it is important that we have +tests that prevent regressions affecting the ability to parse previously-written data.

+ +

The files included represent persisted checkpoints that exercise different +features of thermos. The existing files should not be modified unless +we are accepting backwards incompatibility, such as with a major release.

+ +

It is not practical to write source code to generate these files on the fly, +as source would be vulnerable to drift (e.g. due to refactoring) in ways +that would undermine the goal of ensuring backwards compatibility.

+ +

The most common reason to add a new checkpoint file would be to provide +coverage for new thermos features that alter the data format. This is +accomplished by writing and running a +job configuration that exercises the feature, and +copying the checkpoint file from the sandbox directory, by default this is +/var/run/thermos/checkpoints/<aurora task id>.

+ +

Quick Links

The ASF

+ + + Added: aurora/site/publish/documentation/0.17.0/development/thrift/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/development/thrift/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/development/thrift/index.html (added) +++ aurora/site/publish/documentation/0.17.0/development/thrift/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,184 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Thrift

+ +

Aurora uses Apache Thrift for representing structured data in +client/server RPC protocol as well as for internal data storage. While Thrift is capable of +correctly handling additions and renames of the existing members, field removals must be done +carefully to ensure backwards compatibility and provide predictable deprecation cycle. This +document describes general guidelines for making Thrift schema changes to the existing fields in +api.thrift.

+ +

It is highly recommended to go through the +Thrift: The Missing Guide first to refresh on +basic Thrift schema concepts.

+ +

Checklist

+ +

Every existing Thrift schema modification is unique in its requirements and must be analyzed +carefully to identify its scope and expected consequences. The following checklist may help in that +analysis: +* Is this a new field/struct? If yes, go ahead +* Is this a pure field/struct rename without any type/structure change? If yes, go ahead and rename +* Anything else, read further to make sure your change is properly planned

+ +

Deprecation cycle

+ +

Any time a breaking change (e.g.: field replacement or removal) is required, the following cycle +must be followed:

+ +

vCurrent

+ +

Change is applied in a way that does not break scheduler/client with this version to +communicate with scheduler/client from vCurrent-1. +* Do not remove or rename the old field +* Add a new field as an eventual replacement of the old one and implement a dual read/write +anywhere the old field is used. If a thrift struct is mapped in the DB store make sure both columns +are marked as NOT NULL +* Check storage.thrift to see if +the affected struct is stored in Aurora scheduler storage. If so, it’s almost certainly also +necessary to perform a DB migration. +* Add a deprecation jira ticket into the vCurrent+1 release candidate +* Add a TODO for the deprecated field mentioning the jira ticket

+ +

vCurrent+1

+ +

Finalize the change by removing the deprecated fields from the Thrift schema. +* Drop any dual read/write routines added in the previous version +* Remove thrift backfilling in scheduler +* Remove the deprecated Thrift field

+ +

Testing

+ +

It’s always advisable to test your changes in the local vagrant environment to build more +confidence that you change is backwards compatible. It’s easy to simulate different +client/scheduler versions by playing with aurorabuild command. See this document +for more.

+ +

Quick Links

The ASF

+ + + Added: aurora/site/publish/documentation/0.17.0/development/ui/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/development/ui/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/development/ui/index.html (added) +++ aurora/site/publish/documentation/0.17.0/development/ui/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,174 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Developing the Aurora Scheduler UI

+ +

Installing bower (optional)

+ +

Third party JS libraries used in Aurora (located at 3rdparty/javascript/bower_components) are +managed by bower, a JS dependency manager. Bower is only required if you plan to add, remove or +update JS libraries. Bower can be installed using the following command:

npm install -g bower
+

+ +

Bower depends on node.js and npm. The easiest way to install node on a mac is via brew:

brew install node
+

+ +

For more node.js installation options refer to https://github.com/joyent/node/wiki/Installation.

+ +

More info on installing and using bower can be found at: http://bower.io/. Once installed, you can +use the following commands to view and modify the bower repo at +3rdparty/javascript/bower_components

bower list
+bower install <library name>
+bower remove <library name>
+bower update <library name>
+bower help
+

+ +

Faster Iteration in Vagrant

+ +

The scheduler serves UI assets from the classpath. For production deployments this means the assets +are served from within a jar. However, for faster development iteration, the vagrant image is +configured to add the scheduler subtree of /vagrant/dist/resources/main to the head of +CLASSPATH. This path is configured as a shared filesystem to the path on the host system where +your Aurora repository lives. This means that any updates under dist/resources/main/scheduler in +your checkout will be reflected immediately in the UI served from within the vagrant image.

+ +

The one caveat to this is that this path is under dist not src. This is because the assets must +be processed by gradle before they can be served. So, unfortunately, you cannot just save your local +changes and see them reflected in the UI, you must first run ./gradlew processResources. This is +less than ideal, but better than having to restart the scheduler after every change. Additionally, +gradle makes this process somewhat easier with the use of the --continuous flag. If you run: +./gradlew processResources --continuous gradle will monitor the filesystem for changes and run the +task automatically as necessary. This doesn’t quite provide hot-reload capabilities, but it does +allow for <5s from save to changes being visibile in the UI with no further action required on the +part of the developer.

+ +

Quick Links

The ASF

+ + + Added: aurora/site/publish/documentation/0.17.0/features/constraints/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/features/constraints/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/features/constraints/index.html (added) +++ aurora/site/publish/documentation/0.17.0/features/constraints/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,244 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Scheduling Constraints

+ +

By default, Aurora will pick any random agent with sufficient resources +in order to schedule a task. This scheduling choice can be further +restricted with the help of constraints.

+ +

Mesos Attributes

+ +

Data centers are often organized with hierarchical failure domains. Common failure domains +include hosts, racks, rows, and PDUs. If you have this information available, it is wise to tag +the Mesos agent with them as +attributes.

+ +

The Mesos agent --attributes command line argument can be used to mark agents with +static key/value pairs, so called attributes (not to be confused with --resources, which are +dynamic and accounted).

+ +

For example, consider the host cluster1-aaa-03-sr2 and its following attributes (given in +key:value format): host:cluster1-aaa-03-sr2 and rack:aaa.

+ +

Aurora makes these attributes available for matching with scheduling constraints.

+ +

Limit Constraints

+ +

Limit constraints allow to control machine diversity using constraints. The below +constraint ensures that no more than two instances of your job may run on a single host. +Think of this as a “group by” limit.

Service(
+  name = 'webservice',
+  role = 'www-data',
+  constraints = {
+    'host': 'limit:2',
+  }
+  ...
+)
+

+ +

Likewise, you can use constraints to control rack diversity, e.g. at +most one task per rack:

constraints = {
+  'rack': 'limit:1',
+}
+

+ +

Use these constraints sparingly as they can dramatically reduce Tasks’ schedulability. +Further details are available in the reference documentation on +Scheduling Constraints.

+ +

Value Constraints

+ +

Value constraints can be used to express that a certain attribute with a certain value +should be present on a Mesos agent. For example, the following job would only be +scheduled on nodes that claim to have an SSD as their disk.

Service(
+  name = 'webservice',
+  role = 'www-data',
+  constraints = {
+    'disk': 'SSD',
+  }
+  ...
+)
+

+ +

Further details are available in the reference documentation on +Scheduling Constraints.

+ +

Running stateful services

+ +

Aurora is best suited to run stateless applications, but it also accommodates for stateful services +like databases, or services that otherwise need to always run on the same machines.

+ +

Dedicated attribute

+ +

Most of the Mesos attributes arbitrary and available for custom use. There is one exception, +though: the dedicated attribute. Aurora treats this specially, and only allows matching jobs to +run on these machines, and will only schedule matching jobs on these machines.

+ +

Syntax

+ +

The dedicated attribute has semantic meaning. The format is $role(/.*)?. When a job is created, +the scheduler requires that the $role component matches the role field in the job +configuration, and will reject the job creation otherwise. The remainder of the attribute is +free-form. We’ve developed the idiom of formatting this attribute as $role/$job, but do not +enforce this. For example: a job devcluster/www-data/prod/hello with a dedicated constraint set as +www-data/web.multi will have its tasks scheduled only on Mesos agents configured with: +--attributes=dedicated:www-data/web.multi.

+ +

A wildcard (*) may be used for the role portion of the dedicated attribute, which will allow any +owner to elect for a job to run on the host(s). For example: tasks from both +devcluster/www-data/prod/hello and devcluster/vagrant/test/hello with a dedicated constraint +formatted as */web.multi will be scheduled only on Mesos agents configured with +--attributes=dedicated:*/web.multi. This may be useful when assembling a virtual cluster of +machines sharing the same set of traits or requirements.

+ +

Example

+ +

Consider the following agent command line:

mesos-slave --attributes="dedicated:db_team/redis" ...
+

+ +

And this job configuration:

Service(
+  name = 'redis',
+  role = 'db_team',
+  constraints = {
+    'dedicated': 'db_team/redis'
+  }
+  ...
+)
+

+ +

The job configuration is indicating that it should only be scheduled on agents with the attribute +dedicated:db_team/redis. Additionally, Aurora will prevent any tasks that do not have that +constraint from running on those agents.

+ +

Quick Links

The ASF

+ + + Added: aurora/site/publish/documentation/0.17.0/features/containers/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/features/containers/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/features/containers/index.html (added) +++ aurora/site/publish/documentation/0.17.0/features/containers/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,256 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Containers

+ +

Aurora supports several containerizers, notably the Mesos containerizer and the Docker +containerizer. The Mesos containerizer uses native OS features directly to provide isolation between +containers, while the Docker containerizer delegates container management to the Docker engine.

+ +

The support for launching container images via both containerizers has to be +enabled by a cluster operator.

+ +

Mesos Containerizer

+ +

The Mesos containerizer is the native Mesos containerizer solution. It allows tasks to be +run with an array of pluggable isolators and can launch tasks using +Docker images, +AppC images, or directly on the agent host +filesystem.

+ +

The following example (available in our Vagrant environment) +launches a hello world example within a debian/jessie Docker image:

$ cat /vagrant/examples/jobs/hello_docker_image.aurora
+hello_loop = Process(
+  name = 'hello',
+  cmdline = """
+    while true; do
+      echo hello world
+      sleep 10
+    done
+  """)
+
+task = Task(
+  processes = [hello_loop],
+  resources = Resources(cpu=1, ram=1*MB, disk=8*MB)
+)
+
+jobs = [
+  Service(
+    cluster = 'devcluster',
+    environment = 'devel',
+    role = 'www-data',
+    name = 'hello_docker_image',
+    task = task,
+    container = Mesos(image=DockerImage(name='debian', tag='jessie'))
+  )
+]
+

+ +

Docker and Appc images are designated using an appropriate image property of the Mesos +configuration object. If either container or image is left unspecified, the host filesystem +will be used. Further details of how to specify images can be found in the +Reference Documentation.

+ +

By default, Aurora launches processes as the Linux user named like the used role (e.g. www-data +in the example above). This user has to exist on the host filesystem. If it does not exist within +the container image, it will be created automatically. Otherwise, this user and its primary group +has to exist in the image with matching uid/gid.

+ +

For more information on the Mesos containerizer filesystem, namespace, and isolator features, visit +Mesos Containerizer and +Mesos Container Images.

+ +

Docker Containerizer

+ +

The Docker containerizer launches container images using the Docker engine. It may often provide +more advanced features than the native Mesos containerizer, but has to be installed separately to +Mesos on each agent host.

+ +

Starting with the 0.17.0 release, image can be specified with a {{docker.image[name][tag]}} binder so that +the tag can be resolved to a concrete image digest. This ensures that the job always uses the same image +across restarts, even if the version identified by the tag has been updated, guaranteeing that only job +updates can mutate configuration.

+ +

Example (available in the Vagrant environment):

$ cat /vagrant/examples/jobs/hello_docker_engine.aurora
+hello_loop = Process(
+  name = 'hello',
+  cmdline = """
+    while true; do
+      echo hello world
+      sleep 10
+    done
+  """)
+
+task = Task(
+  processes = [hello_loop],
+  resources = Resources(cpu=1, ram=1*MB, disk=8*MB)
+)
+
+jobs = [
+  Service(
+    cluster = 'devcluster',
+    environment = 'devel',
+    role = 'www-data',
+    name = 'hello_docker',
+    task = task,
+    container = Docker(image = 'python:2.7')
+  ), Service(
+    cluster = 'devcluster',
+    environment = 'devel',
+    role = 'www-data',
+    name = 'hello_docker_engine_binding',
+    task = task,
+    container = Docker(image = '{{docker.image[library/python][2.7]}}')
+  )
+]
+

+ +

Note, this feature requires a v2 Docker registry. If using a private Docker registry its url +must be specified in the clusters.json configuration file under the key docker_registry. +If not specified docker_registry defaults to https://registry-1.docker.io (Docker Hub).

+ +

Example: + # clusters.json + [{ + “name”: “devcluster”, + … + “docker_registry”: “https://registry.example.com” + }]

+ +

Details of how to use Docker via the Docker engine can be found in the +Reference Documentation. Please note that in order to +correctly execute processes inside a job, the Docker container must have Python 2.7 and potentitally +further Mesos dependencies installed. This limitation does not hold for Docker containers used via +the Mesos containerizer.

+ +

For more information on launching Docker containers through the Docker containerizer, visit +Docker Containerizer

+ +

Quick Links

The ASF

+ + + Added: aurora/site/publish/documentation/0.17.0/features/cron-jobs/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.17.0/features/cron-jobs/index.html?rev=1783940&view=auto ============================================================================== --- aurora/site/publish/documentation/0.17.0/features/cron-jobs/index.html (added) +++ aurora/site/publish/documentation/0.17.0/features/cron-jobs/index.html Tue Feb 21 20:54:58 2017 @@ -0,0 +1,269 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Cron Jobs

+ +

Aurora supports execution of scheduled jobs on a Mesos cluster using cron-style syntax.

+ +

Overview
Collision Policies
Failure recovery
Interacting with cron jobs via the Aurora CLI + +
- cron schedule
- cron deschedule
- cron start
- job killall, job restart, job kill
Technical Note About Syntax
Caveats + +

+ +

Overview

+ +

A job is identified as a cron job by the presence of a +cron_schedule attribute containing a cron-style schedule in the +Job object. Examples of cron schedules +include “every 5 minutes” (*/5 * * * *), “Fridays at 17:00” (* 17 * * FRI), and +“the 1st and 15th day of the month at 03:00” (0 3 1,15 *).

+ +

Example (available in the Vagrant environment):

$ cat /vagrant/examples/jobs/cron_hello_world.aurora
+# A cron job that runs every 5 minutes.
+jobs = [
+  Job(
+    cluster = 'devcluster',
+    role = 'www-data',
+    environment = 'test',
+    name = 'cron_hello_world',
+    cron_schedule = '*/5 * * * *',
+    task = SimpleTask(
+      'cron_hello_world',
+      'echo "Hello world from cron, the time is now $(date --rfc-822)"'),
+  ),
+]
+

+ +

Collision Policies

+ +

The cron_collision_policy field specifies the scheduler’s behavior when a new cron job is +triggered while an older run hasn’t finished. The scheduler has two policies available:

+ +

KILL_EXISTING: The default policy - on a collision the old instances are killed and a instances with the current +configuration are started.
CANCEL_NEW: On a collision the new run is cancelled.

+ +

Note that the use of CANCEL_NEW is likely a code smell - interrupted cron jobs should be able +to recover their progress on a subsequent invocation, otherwise they risk having their work queue +grow faster than they can process it.

+ +

Failure recovery

+ +

Unlike with services, which aurora will always re-execute regardless of exit status, instances of +cron jobs retry according to the max_task_failures attribute of the +Task object. To get “run-until-success” semantics, +set max_task_failures to -1.

+ +

Interacting with cron jobs via the Aurora CLI

+ +

Most interaction with cron jobs takes place using the cron subcommand. See aurora cron -h +for up-to-date usage instructions.

+ +

cron schedule

+ +

Schedules a new cron job on the Aurora cluster for later runs or replaces the existing cron template +with a new one. Only future runs will be affected, any existing active tasks are left intact.

$ aurora cron schedule devcluster/www-data/test/cron_hello_world /vagrant/examples/jobs/cron_hello_world.aurora
+

+ +

cron deschedule

+ +

Deschedules a cron job, preventing future runs but allowing current runs to complete.

$ aurora cron deschedule devcluster/www-data/test/cron_hello_world
+

+ +

cron start

+ +

Start a cron job immediately, outside of its normal cron schedule.

$ aurora cron start devcluster/www-data/test/cron_hello_world
+

+ +

job killall, job restart, job kill

+ +

Cron jobs create instances running on the cluster that you can interact with like normal Aurora +tasks with job kill and job restart.

+ +

Technical Note About Syntax

+ +

cron_schedule uses a restricted subset of BSD crontab syntax. While the +execution engine currently uses Quartz, the schedule parsing is custom, a subset of FreeBSD +crontab(5) syntax. See +the source +for details.

+ +

Caveats

+ +

Failovers

+ +

No failover recovery. Aurora does not record the latest minute it fired +triggers for across failovers. Therefore it’s possible to miss triggers +on failover. Note that this behavior may change in the future.

+ +

It’s necessary to sync time between schedulers with something like ntpd. +Clock skew could cause double or missed triggers in the case of a failover.

+ +

Collision policy is best-effort

+ +

Aurora aims to always have at least one copy of a given instance running at a time - it’s +an AP system, meaning it chooses Availability and Partition Tolerance at the expense of +Consistency.

+ +

If your collision policy was CANCEL_NEW and a task has terminated but +Aurora has not noticed this Aurora will go ahead and create your new +task.

+ +

If your collision policy was KILL_EXISTING and a task was marked LOST +but not yet GCed Aurora will go ahead and create your new task without +attempting to kill the old one (outside the GC interval).

+ +

Timezone Configuration

+ +

Cron timezone is configured indepdendently of JVM timezone with the -cron_timezone flag and +defaults to UTC.

+ +

Quick Links

The ASF

+ + +

Documentation + + + 0.17.0 + (latest) + + + 0.16.0 + + + 0.15.0 + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Design Documents

Quick Links

The ASF

Documentation + + + 0.17.0 + (latest) + + + 0.16.0 + + + 0.15.0 + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Command Hooks for the Aurora Client

Introduction/Motivation

Registering Hooks

Global Hooks

The API

Skipping Hooks

Quick Links

The ASF

Documentation + + + 0.17.0 + (latest) + + + 0.16.0 + + + 0.15.0 + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Developing the Aurora Scheduler

Prerequisite

Getting Started

Running the Tests

Running the build with code quality checks

Running integration tests

Creating a bundle for deployment

Developing Aurora Java code

Setting up an IDE

Adding or Upgrading a Dependency

Developing the Aurora Build System

Bootstrapping Gradle

Quick Links

The ASF

Documentation + + + 0.17.0 + (latest) + + + 0.16.0 + + + 0.15.0 + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Python Build Conventions

Annotated example - apache.thermos.runner

Thermos Test resources

Quick Links

The ASF

Documentation + + + 0.17.0 + (latest) + + + 0.16.0 + + + 0.15.0 + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Thrift

Checklist

Deprecation cycle

vCurrent

vCurrent+1

Testing

Quick Links

The ASF

Documentation + + + 0.17.0 + (latest) + + + 0.16.0 + + + 0.15.0 + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Developing the Aurora Scheduler UI

Installing bower (optional)

Faster Iteration in Vagrant

Quick Links

The ASF

Documentation + + + 0.17.0 + (latest) + + + 0.16.0 + + + 0.15.0 + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Scheduling Constraints

Mesos Attributes

Limit Constraints

Value Constraints

Running stateful services

Dedicated attribute

Syntax

Example

Quick Links

The ASF

Documentation + + + 0.17.0 + (latest) + + + 0.16.0 + + + 0.15.0 + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Containers

Mesos Containerizer

Docker Containerizer

Quick Links

The ASF

Documentation + + + 0.17.0 + (latest) + + + 0.16.0 + + + 0.15.0 + + + 0.14.0 + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Cron Jobs

Overview

Collision Policies

Failure recovery

Interacting with cron jobs via the Aurora CLI

cron schedule

cron deschedule

cron start

job killall, job restart, job kill

Technical Note About Syntax

Caveats

Failovers

Collision policy is best-effort

Documentation + +

Documentation + +

Documentation + +

Documentation + +

Documentation + +

Documentation + +

Documentation + +

Documentation + +

Documentation + +