; Tue, 14 Jun 2016 21:35:33 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1748470 [6/19] - in /aurora/site: data/ publish/ publish/blog/ publish/blog/aurora-0-14-0-released/ publish/documentation/0.10.0/ publish/documentation/0.10.0/build-system/ publish/documentation/0.10.0/client-cluster-configuration/ publish... Date: Tue, 14 Jun 2016 21:35:30 -0000 To: commits@aurora.apache.org From: serb@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20160614213533.4D7D63A039A@svn01-us-west.apache.org> archived-at: Tue, 14 Jun 2016 21:35:46 -0000 Added: aurora/site/publish/documentation/0.14.0/getting-started/tutorial/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/getting-started/tutorial/index.html?rev=1748470&view=auto ============================================================================== --- aurora/site/publish/documentation/0.14.0/getting-started/tutorial/index.html (added) +++ aurora/site/publish/documentation/0.14.0/getting-started/tutorial/index.html Tue Jun 14 21:35:25 2016 @@ -0,0 +1,375 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Aurora Tutorial

+ +

This tutorial shows how to use the Aurora scheduler to run (and “printf-debug”) +a hello world program on Mesos. This is the recommended document for new Aurora users +to start getting up to speed on the system.

+ +

Prerequisite
The Script
Aurora Configuration
Creating the Job
Watching the Job Run
Cleanup
Next Steps

+ +

Prerequisite

+ +

This tutorial assumes you are running Aurora locally using Vagrant. +However, in general the instructions are also applicable to any other +Aurora installation.

+ +

Unless otherwise stated, all commands are to be run from the root of the aurora +repository clone.

+ +

The Script

+ +

Our “hello world” application is a simple Python script that loops +forever, displaying the time every few seconds. Copy the code below and +put it in a file named hello_world.py in the root of your Aurora repository clone +(Note: this directory is the same as /vagrant inside the Vagrant VMs).

+ +

The script has an intentional bug, which we will explain later on.

+ + +

import time
+
+def main():
+  SLEEP_DELAY = 10
+  # Python experts - ignore this blatant bug.
+  for i in xrang(100):
+    print("Hello world! The time is now: %s. Sleeping for %d secs" % (
+      time.asctime(), SLEEP_DELAY))
+    time.sleep(SLEEP_DELAY)
+
+if __name__ == "__main__":
+  main()
+

+ +

Aurora Configuration

+ +

Once we have our script/program, we need to create a configuration +file that tells Aurora how to manage and launch our Job. Save the below +code in the file hello_world.aurora.

+ + +

pkg_path = '/vagrant/hello_world.py'
+
+# we use a trick here to make the configuration change with
+# the contents of the file, for simplicity.  in a normal setting, packages would be
+# versioned, and the version number would be changed in the configuration.
+import hashlib
+with open(pkg_path, 'rb') as f:
+  pkg_checksum = hashlib.md5(f.read()).hexdigest()
+
+# copy hello_world.py into the local sandbox
+install = Process(
+  name = 'fetch_package',
+  cmdline = 'cp %s . && echo %s && chmod +x hello_world.py' % (pkg_path, pkg_checksum))
+
+# run the script
+hello_world = Process(
+  name = 'hello_world',
+  cmdline = 'python -u hello_world.py')
+
+# describe the task
+hello_world_task = SequentialTask(
+  processes = [install, hello_world],
+  resources = Resources(cpu = 1, ram = 1*MB, disk=8*MB)
 )
+
+jobs = [
+  Service(cluster = 'devcluster',
+          environment = 'devel',
+          role = 'www-data',
+          name = 'hello_world',
+          task = hello_world_task)
+]
+

+ +

There is a lot going on in that configuration file:

+ +

From a “big picture” viewpoint, it first defines two +Processes. Then it defines a Task that runs the two Processes in the +order specified in the Task definition, as well as specifying what +computational and memory resources are available for them. Finally, +it defines a Job that will schedule the Task on available and suitable +machines. This Job is the sole member of a list of Jobs; you can +specify more than one Job in a config file.
At the Process level, it specifies how to get your code into the +local sandbox in which it will run. It then specifies how the code is +actually run once the second Process starts.

+ +

For more about Aurora configuration files, see the Configuration +Tutorial and the Configuration +Reference (preferably after finishing this +tutorial).

+ +

Creating the Job

+ +

We’re ready to launch our job! To do so, we use the Aurora Client to +issue a Job creation request to the Aurora scheduler.

+ +

Many Aurora Client commands take a job key argument, which uniquely +identifies a Job. A job key consists of four parts, each separated by a +“/”. The four parts are <cluster>/<role>/<environment>/<jobname> +in that order:

+ +

Cluster refers to the name of a particular Aurora installation.
Role names are user accounts existing on the agent machines. If you +don’t know what accounts are available, contact your sysadmin.
Environment names are namespaces; you can count on test, devel, +staging and prod existing.
Jobname is the custom name of your job.

+ +

When comparing two job keys, if any of the four parts is different from +its counterpart in the other key, then the two job keys identify two separate +jobs. If all four values are identical, the job keys identify the same job.

+ +

The clusters.json client configuration +for the Aurora scheduler defines the available cluster names. +For Vagrant, from the top-level of your Aurora repository clone, do:

$ vagrant ssh
+

+ +

Followed by:

vagrant@aurora:~$ cat /etc/aurora/clusters.json
+

+ +

You’ll see something like the following. The name value shown here, corresponds to a job key’s cluster value.

[{
+  "name": "devcluster",
+  "zk": "192.168.33.7",
+  "scheduler_zk_path": "/aurora/scheduler",
+  "auth_mechanism": "UNAUTHENTICATED",
+  "slave_run_directory": "latest",
+  "slave_root": "/var/lib/mesos"
+}]
+

+ +

The Aurora Client command that actually runs our Job is aurora job create. It creates a Job as +specified by its job key and configuration file arguments and runs it.

aurora job create <cluster>/<role>/<environment>/<jobname> <config_file>
+

+ +

Or for our example:

aurora job create devcluster/www-data/devel/hello_world /vagrant/hello_world.aurora
+

+ +

After entering our virtual machine using vagrant ssh, this returns:

vagrant@aurora:~$ aurora job create devcluster/www-data/devel/hello_world /vagrant/hello_world.aurora
+ INFO] Creating job hello_world
+ INFO] Checking status of devcluster/www-data/devel/hello_world
+Job create succeeded: job url=http://aurora.local:8081/scheduler/www-data/devel/hello_world
+

+ +

Watching the Job Run

+ +

Now that our job is running, let’s see what it’s doing. Access the +scheduler web interface at http://$scheduler_hostname:$scheduler_port/scheduler +Or when using vagrant, http://192.168.33.7:8081/scheduler +First we see what Jobs are scheduled:

+ +

Scheduled Jobs

+ +

Click on your user name, which in this case was www-data, and we see the Jobs associated +with that role:

+ +

Role Jobs

+ +

If you click on your hello_world Job, you’ll see:

+ +

hello_world Job

+ +

Oops, looks like our first job didn’t quite work! The task is temporarily throttled for +having failed on every attempt of the Aurora scheduler to run it. We have to figure out +what is going wrong.

+ +

On the Completed tasks tab, we see all past attempts of the Aurora scheduler to run our job.

+ +

Completed tasks tab

+ +

We can navigate to the Task page of a failed run by clicking on the host link.

+ +

Task page

+ +

Once there, we see that the hello_world process failed. The Task page +captures the standard error and standard output streams and makes them available. +Clicking through to stderr on the failed hello_world process, we see what happened.

+ +

stderr page

+ +

It looks like we made a typo in our Python script. We wanted xrange, +not xrang. Edit the hello_world.py script to use the correct function +and save it as hello_world_v2.py. Then update the hello_world.aurora +configuration to the newest version.

+ +

In order to try again, we can now instruct the scheduler to update our job:

vagrant@aurora:~$ aurora update start devcluster/www-data/devel/hello_world /vagrant/hello_world.aurora
+ INFO] Starting update for: hello_world
+Job update has started. View your update progress at http://aurora.local:8081/scheduler/www-data/devel/hello_world/update/8ef38017-e60f-400d-a2f2-b5a8b724e95b
+

+ +

This time, the task comes up.

+ +

Running Job

+ +

By again clicking on the host, we inspect the Task page, and see that the +hello_world process is running.

+ +

Running Task page

+ +

We then inspect the output by clicking on stdout and see our process’ +output:

+ +

stdout page

+ +

Cleanup

+ +

Now that we’re done, we kill the job using the Aurora client:

vagrant@aurora:~$ aurora job killall devcluster/www-data/devel/hello_world
+ INFO] Killing tasks for job: devcluster/www-data/devel/hello_world
+ INFO] Instances to be killed: [0]
+Successfully killed instances [0]
+Job killall succeeded
+

+ +

The job page now shows the hello_world tasks as completed.

+ +

Killed Task page

+ +

Next Steps

+ +

Now that you’ve finished this Tutorial, you should read or do the following:

+ +

The Aurora Configuration Tutorial, which provides more examples +and best practices for writing Aurora configurations. You should also look at +the Aurora Configuration Reference.
Explore the Aurora Client - use aurora -h, and read the +Aurora Client Commands document.

+ +

Quick Links

The ASF

Copyright 2014 Apache Software Foundation. Licensed under the Apache License v2.0. The Aurora Borealis IX photo displayed on the homepage is available under a Creative Commons BY-NC-ND 2.0 license. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.

+ + + Propchange: aurora/site/publish/documentation/0.14.0/getting-started/tutorial/index.html ------------------------------------------------------------------------------ svn:eol-style = native Propchange: aurora/site/publish/documentation/0.14.0/getting-started/tutorial/index.html ------------------------------------------------------------------------------ svn:keywords = Id Propchange: aurora/site/publish/documentation/0.14.0/getting-started/tutorial/index.html ------------------------------------------------------------------------------ svn:mime-type = text/html Added: aurora/site/publish/documentation/0.14.0/getting-started/vagrant/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/getting-started/vagrant/index.html?rev=1748470&view=auto ============================================================================== --- aurora/site/publish/documentation/0.14.0/getting-started/vagrant/index.html (added) +++ aurora/site/publish/documentation/0.14.0/getting-started/vagrant/index.html Tue Jun 14 21:35:25 2016 @@ -0,0 +1,263 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

A local Cluster with Vagrant

+ +

This document shows you how to configure a complete cluster using a virtual machine. This setup +replicates a real cluster in your development machine as closely as possible. After you complete +the steps outlined here, you will be ready to create and run your first Aurora job.

+ +

The following sections describe these steps in detail:

+ +

Overview
Install VirtualBox and Vagrant
Clone the Aurora repository
Start the local cluster
Log onto the VM
Run your first job
Rebuild components
Shut down or delete your local cluster
Troubleshooting

+ +

Overview

+ +

The Aurora distribution includes a set of scripts that enable you to create a local cluster in +your development machine. These scripts use Vagrant and +VirtualBox to run and configure a virtual machine. Once the +virtual machine is running, the scripts install and initialize Aurora and any required components +to create the local cluster.

+ +

Install VirtualBox and Vagrant

+ +

First, download and install VirtualBox on your development machine.

+ +

Then download and install Vagrant. To verify that the installation +was successful, open a terminal window and type the vagrant command. You should see a list of +common commands for this tool.

+ +

Clone the Aurora repository

+ +

To obtain the Aurora source distribution, clone its Git repository using the following command:

 git clone git://git.apache.org/aurora.git
+

+ +

Start the local cluster

+ +

Now change into the aurora/ directory, which contains the Aurora source code and +other scripts and tools:

 cd aurora/
+

+ +

To start the local cluster, type the following command:

 vagrant up
+

+ +

This command uses the configuration scripts in the Aurora distribution to:

+ +

Download a Linux system image.
Start a virtual machine (VM) and configure it.
Install the required build tools on the VM.
Install Aurora’s requirements (like Mesos and +Zookeeper) on the VM.
Build and install Aurora from source on the VM.
Start Aurora’s services on the VM.

+ +

This process takes several minutes to complete.

+ +

You may notice a warning that guest additions in the VM don’t match your version of VirtualBox. +This should generally be harmless, but you may wish to install a vagrant plugin to take care of +mismatches like this for you:

 vagrant plugin install vagrant-vbguest
+

+ +

With this plugin installed, whenever you vagrant up the plugin will upgrade the guest additions +for you when a version mis-match is detected. You can read more about the plugin +here.

+ +

To verify that Aurora is running on the cluster, visit the following URLs:

+ +

Scheduler - http://192.168.33.7:8081
Observer - http://192.168.33.7:1338
Mesos Master - http://192.168.33.7:5050
Mesos Agent - http://192.168.33.7:5051

+ +

Log onto the VM

+ +

To SSH into the VM, run the following command in your development machine:

 vagrant ssh
+

+ +

To verify that Aurora is installed in the VM, type the aurora command. You should see a list +of arguments and possible commands.

+ +

The /vagrant directory on the VM is mapped to the aurora/ local directory +from which you started the cluster. You can edit files inside this directory in your development +machine and access them from the VM under /vagrant.

+ +

A pre-installed clusters.json file refers to your local cluster as devcluster, which you +will use in client commands.

+ +

Run your first job

+ +

Now that your cluster is up and running, you are ready to define and run your first job in Aurora. +For more information, see the Aurora Tutorial.

+ +

Rebuild components

+ +

If you are changing Aurora code and would like to rebuild a component, you can use the aurorabuild +command on the VM to build and restart a component. This is considerably faster than destroying +and rebuilding your VM.

+ +

aurorabuild accepts a list of components to build and update. To get a list of supported +components, invoke the aurorabuild command with no arguments:

 vagrant ssh -c 'aurorabuild client'
+

+ +

Shut down or delete your local cluster

+ +

To shut down your local cluster, run the vagrant halt command in your development machine. To +start it again, run the vagrant up command.

+ +

Once you are finished with your local cluster, or if you would otherwise like to start from scratch, +you can use the command vagrant destroy to turn off and delete the virtual file system.

+ +

Troubleshooting

+ +

Most of the Vagrant related problems can be fixed by the following steps:

+ +

Destroying the vagrant environment with vagrant destroy
Killing any orphaned VMs (see AURORA-499) with virtualbox UI or VBoxManage command line tool
Cleaning the repository of build artifacts and other intermediate output with git clean -fdx
Bringing up the vagrant environment with vagrant up

+ +

If that still doesn’t solve your problem, make sure to inspect the log files:

+ +

Scheduler: /var/log/upstart/aurora-scheduler.log
Observer: /var/log/upstart/aurora-thermos-observer.log
Mesos Master: /var/log/mesos/mesos-master.INFO (also see .WARNING and .ERROR)
Mesos Agent: /var/log/mesos/mesos-slave.INFO (also see .WARNING and .ERROR)

+ +

Quick Links

The ASF

+ + + Propchange: aurora/site/publish/documentation/0.14.0/getting-started/vagrant/index.html ------------------------------------------------------------------------------ svn:eol-style = native Propchange: aurora/site/publish/documentation/0.14.0/getting-started/vagrant/index.html ------------------------------------------------------------------------------ svn:keywords = Id Propchange: aurora/site/publish/documentation/0.14.0/getting-started/vagrant/index.html ------------------------------------------------------------------------------ svn:mime-type = text/html Added: aurora/site/publish/documentation/0.14.0/images/CPUavailability.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/CPUavailability.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/CPUavailability.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/CompletedTasks.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/CompletedTasks.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/CompletedTasks.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/HelloWorldJob.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/HelloWorldJob.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/HelloWorldJob.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/RoleJobs.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/RoleJobs.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/RoleJobs.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/RunningJob.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/RunningJob.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/RunningJob.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/ScheduledJobs.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/ScheduledJobs.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/ScheduledJobs.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/TaskBreakdown.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/TaskBreakdown.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/TaskBreakdown.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/aurora_hierarchy.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/aurora_hierarchy.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/aurora_hierarchy.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/aurora_logo.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/aurora_logo.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/aurora_logo.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/components.odg URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/components.odg?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/components.odg ------------------------------------------------------------------------------ svn:mime-type = application/vnd.oasis.opendocument.graphics Added: aurora/site/publish/documentation/0.14.0/images/components.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/components.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/components.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/debug-client-test.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/debug-client-test.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/debug-client-test.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/debugging-client-test.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/debugging-client-test.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/debugging-client-test.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/killedtask.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/killedtask.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/killedtask.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/lifeofatask.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/lifeofatask.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/lifeofatask.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/02_19_2015_aurora_adopters_panel_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/02_19_2015_aurora_at_tellapart_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/02_19_2015_aurora_at_twitter_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/02_28_2015_apache_aurora_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/02_28_2015_apache_aurora_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/02_28_2015_apache_aurora_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/03_07_2015_aurora_mesos_in_practice_at_twitter_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/03_25_2014_introduction_to_aurora_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/04_30_2015_monolith_to_microservices_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/08_21_2014_past_present_future_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/08_21_2014_past_present_future_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/08_21_2014_past_present_future_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/09_20_2015_shipping_code_with_aurora_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/09_20_2015_twitter_production_scale_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/10_08_2015_mesos_aurora_on_a_small_scale_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/presentations/10_08_2015_sla_aware_maintenance_for_operators_thumb.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/runningtask.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/runningtask.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/runningtask.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/stderr.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/stderr.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/stderr.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/stdout.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/stdout.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/stdout.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/images/storage_hierarchy.png URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/images/storage_hierarchy.png?rev=1748470&view=auto ============================================================================== Binary file - no diff available. Propchange: aurora/site/publish/documentation/0.14.0/images/storage_hierarchy.png ------------------------------------------------------------------------------ svn:mime-type = image/png Added: aurora/site/publish/documentation/0.14.0/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/index.html?rev=1748470&view=auto ============================================================================== --- aurora/site/publish/documentation/0.14.0/index.html (added) +++ aurora/site/publish/documentation/0.14.0/index.html Tue Jun 14 21:35:25 2016 @@ -0,0 +1,216 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Introduction

+ +

Apache Aurora is a service scheduler that runs on top of Apache Mesos, enabling you to run +long-running services, cron jobs, and ad-hoc jobs that take advantage of Apache Mesos’ scalability, +fault-tolerance, and resource isolation.

+ +

We encourage you to ask questions on the Aurora user list or +the #aurora IRC channel on irc.freenode.net.

+ +

Getting Started

+ +

Information for everyone new to Apache Aurora.

+ +

Features

+ +

Description of important Aurora features.

+ +

Operators

+ +

For those that wish to manage and fine-tune an Aurora cluster.

+ +

Reference

+ +

The complete reference of commands, configuration options, and scheduler internals.

+ +

Task lifecycle
Configuration (.aurora files) + +
Aurora Client + +
- Client Commands
- Client Hooks
- Client Cluster Configuration
Scheduler Configuration

+ +

Additional Resources

+ +

Developers

+ +

All the information you need to start modifying Aurora and contributing back to the project.

+ +

Contributing to the project
Committer’s Guide
Design Documents
Developing the Aurora components: + +
- Client
- Scheduler
- Scheduler UI
- Thermos
- Thrift structures

+ +

Quick Links

The ASF

+ + + Propchange: aurora/site/publish/documentation/0.14.0/index.html ------------------------------------------------------------------------------ svn:eol-style = native Propchange: aurora/site/publish/documentation/0.14.0/index.html ------------------------------------------------------------------------------ svn:keywords = Id Propchange: aurora/site/publish/documentation/0.14.0/index.html ------------------------------------------------------------------------------ svn:mime-type = text/html Added: aurora/site/publish/documentation/0.14.0/operations/backup-restore/index.html URL: http://svn.apache.org/viewvc/aurora/site/publish/documentation/0.14.0/operations/backup-restore/index.html?rev=1748470&view=auto ============================================================================== --- aurora/site/publish/documentation/0.14.0/operations/backup-restore/index.html (added) +++ aurora/site/publish/documentation/0.14.0/operations/backup-restore/index.html Tue Jun 14 21:35:25 2016 @@ -0,0 +1,218 @@ + + + + + + Apache Aurora + + + + + + +

+ +

Documentation + +

Recovering from a Scheduler Backup

+ +

Be sure to read the entire page before attempting to restore from a backup, as it may have +unintended consequences.

+ +

Summary

+ +

The restoration procedure replaces the existing (possibly corrupted) Mesos replicated log with an +earlier, backed up, version and requires all schedulers to be taken down temporarily while +restoring. Once completed, the scheduler state resets to what it was when the backup was created. +This means any jobs/tasks created or updated after the backup are unknown to the scheduler and will +be killed shortly after the cluster restarts. All other tasks continue operating as normal.

+ +

Usually, it is a bad idea to restore a backup that is not extremely recent (i.e. older than a few +hours). This is because the scheduler will expect the cluster to look exactly as the backup does, +so any tasks that have been rescheduled since the backup was taken will be killed.

+ +

Instructions below have been verified in Vagrant environment and with minor +syntax/path changes should be applicable to any Aurora cluster.

+ +

Preparation

+ +

Follow these steps to prepare the cluster for restoring from a backup:

+ +

Stop all scheduler instances
Consider blocking external traffic on a port defined in -http_port for all schedulers to +prevent users from interacting with the scheduler during the restoration process. This will help +troubleshooting by reducing the scheduler log noise and prevent users from making changes that will +be erased after the backup snapshot is restored.
Configure aurora_admin access to run all commands listed in +Restore from backup section locally on the leading scheduler:
+ +
- Make sure the clusters.json file configured to +access scheduler directly. Set scheduler_uri setting and remove zk. Since leader can get +re-elected during the restore steps, consider doing it on all scheduler replicas.
- Depending on your particular security approach you will need to either turn off scheduler +authorization by removing scheduler -http_authentication_mechanism flag or make sure the +direct scheduler access is properly authorized. E.g.: in case of Kerberos you will need to make +a /etc/hosts file change to match your local IP to the scheduler URL configured in keytabs:
  + +
Next steps are required to put scheduler into a partially disabled state where it would still be +able to accept storage recovery requests but unable to schedule or change task states. This may be +accomplished by updating the following scheduler configuration options:
+ +
- Set -mesos_master_address to a non-existent zk address. This will prevent scheduler from +registering with Mesos. E.g.: -mesos_master_address=zk://localhost:1111/mesos/master
- -max_registration_delay - set to sufficiently long interval to prevent registration timeout +and as a result scheduler suicide. E.g: -max_registration_delay=360mins
- Make sure -reconciliation_initial_delay option is set high enough (e.g.: 365days) to +prevent accidental task GC. This is important as scheduler will attempt to reconcile the cluster +state and will kill all tasks when restarted with an empty Mesos replicated log.
Restart all schedulers

+ +

Cleanup and re-initialize Mesos replicated log

+ +

Get rid of the corrupted files and re-initialize Mesos replicated log:

+ +

Stop schedulers
Delete all files under -native_log_file_path on all schedulers
Initialize Mesos replica’s log file: sudo mesos-log initialize --path=<-native_log_file_path>
Start schedulers

+ +

Restore from backup

+ +

At this point the scheduler is ready to rehydrate from the backup:

+ +

Identify the leading scheduler by:
+ +
- examining the scheduler_lifecycle_LEADER_AWAITING_REGISTRATION metric at the scheduler +/vars endpoint. Leader will have 1. All other replicas - 0.
- examining scheduler logs
- or examining Zookeeper registration under the path defined by -zk_endpoints +and -serverset_path
Locate the desired backup file, copy it to the leading scheduler’s -backup_dir folder and stage +recovery by running the following command on a leader +aurora_admin scheduler_stage_recovery --bypass-leader-redirect <cluster> scheduler-backup-<yyyy-MM-dd-HH-mm>
At this point, the recovery snapshot is staged and available for manual verification/modification +via aurora_admin scheduler_print_recovery_tasks --bypass-leader-redirect and +scheduler_delete_recovery_tasks --bypass-leader-redirect commands. +See aurora_admin help <command> for usage details.
Commit recovery. This instructs the scheduler to overwrite the existing Mesos replicated log with +the provided backup snapshot and initiate a mandatory failover +aurora_admin scheduler_commit_recovery --bypass-leader-redirect <cluster>

+ +

Cleanup

+ +

Undo any modification done during Preparation sequence.

+ +

Quick Links

The ASF

+ + + Propchange: aurora/site/publish/documentation/0.14.0/operations/backup-restore/index.html ------------------------------------------------------------------------------ svn:eol-style = native Propchange: aurora/site/publish/documentation/0.14.0/operations/backup-restore/index.html ------------------------------------------------------------------------------ svn:keywords = Id Propchange: aurora/site/publish/documentation/0.14.0/operations/backup-restore/index.html ------------------------------------------------------------------------------ svn:mime-type = text/html

Documentation + + + 0.14.0 + (latest) + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Aurora Tutorial

Prerequisite

The Script

Aurora Configuration

Creating the Job

Watching the Job Run

Cleanup

Next Steps

Quick Links

The ASF

Documentation + + + 0.14.0 + (latest) + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

A local Cluster with Vagrant

Overview

Install VirtualBox and Vagrant

Clone the Aurora repository

Start the local cluster

Log onto the VM

Run your first job

Rebuild components

Shut down or delete your local cluster

Troubleshooting

Quick Links

The ASF

Documentation + + + 0.14.0 + (latest) + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Introduction

Getting Started

Features

Operators

Reference

Additional Resources

Developers

Quick Links

The ASF

Documentation + + + 0.14.0 + (latest) + + + 0.13.0 + + + 0.12.0 + + + 0.11.0 + + + 0.10.0 + + + 0.9.0 + + + 0.8.0 + + + 0.7.0-incubating + + + 0.6.0-incubating + + + 0.5.0-incubating + + +

Recovering from a Scheduler Backup

Summary

Preparation

Cleanup and re-initialize Mesos replicated log

Restore from backup

Cleanup

Quick Links

The ASF

Documentation + +

Documentation + +

Documentation + +

Documentation + +