aurora-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jco...@apache.org
Subject aurora git commit: Moving custom executors documentation to features, adding gorealis to tools
Date Thu, 18 Aug 2016 19:08:39 GMT
Repository: aurora
Updated Branches:
  refs/heads/master a4fdf284d -> 370813f11


Moving custom executors documentation to features, adding gorealis to tools

Moved custom executor documentation from docs/operations/configuration.md since it grew out
of
proportion.

Added an explaination on how a thrift object needs to be configured in a createJob call and
referenced using a custom Client for now.

Reviewed at https://reviews.apache.org/r/51192/


Project: http://git-wip-us.apache.org/repos/asf/aurora/repo
Commit: http://git-wip-us.apache.org/repos/asf/aurora/commit/370813f1
Tree: http://git-wip-us.apache.org/repos/asf/aurora/tree/370813f1
Diff: http://git-wip-us.apache.org/repos/asf/aurora/diff/370813f1

Branch: refs/heads/master
Commit: 370813f11e45e9ed0e6237f622dc8881507fca92
Parents: a4fdf28
Author: Renan DelValle <rdelval1@binghamton.edu>
Authored: Thu Aug 18 14:08:14 2016 -0500
Committer: Joshua Cohen <jcohen@apache.org>
Committed: Thu Aug 18 14:08:14 2016 -0500

----------------------------------------------------------------------
 docs/additional-resources/tools.md |   3 +
 docs/features/custom-executors.md  | 149 ++++++++++++++++++++++++++++++++
 docs/operations/configuration.md   | 142 ++----------------------------
 3 files changed, 157 insertions(+), 137 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/aurora/blob/370813f1/docs/additional-resources/tools.md
----------------------------------------------------------------------
diff --git a/docs/additional-resources/tools.md b/docs/additional-resources/tools.md
index 64a4f74..678e550 100644
--- a/docs/additional-resources/tools.md
+++ b/docs/additional-resources/tools.md
@@ -19,3 +19,6 @@ Various tools integrate with Aurora. Is there a tool missing? Let us know,
or su
 
 * Packaging and deployment:
   - [aurora-packaging](https://github.com/apache/aurora-packaging), the source of the official
Aurora packages
+
+* Thrift Clients:
+  - [gorealis](https://github.com/rdelval/gorealis) for communicating with the scheduler
using Go

http://git-wip-us.apache.org/repos/asf/aurora/blob/370813f1/docs/features/custom-executors.md
----------------------------------------------------------------------
diff --git a/docs/features/custom-executors.md b/docs/features/custom-executors.md
new file mode 100644
index 0000000..7438938
--- /dev/null
+++ b/docs/features/custom-executors.md
@@ -0,0 +1,149 @@
+Custom Executors
+================
+
+The configuration file must be a valid **JSON array** and contain, at minimum,
+one executor configuration including the name, command and resources fields and
+must be pointed to by the `-custom_executor_config` flag when the scheduler is
+started.
+
+### Array Entry
+
+Property                 | Description
+-----------------------  | ---------------------------------
+executor (required)      | Description of executor.
+task_prefix (required) ) | Prefix given to tasks launched with this executor's configuration.
+volume_mounts (optional) | Volumes to be mounted in container running executor.
+
+#### executor
+
+Property                 | Description
+-----------------------  | ---------------------------------
+name (required)          | Name of the executor.
+command (required)       | How to run the executor.
+resources (required)     | Overhead to use for each executor instance.
+
+#### command
+
+Property                 | Description
+-----------------------  | ---------------------------------
+value (required)         | The command to execute.
+arguments (optional)     | A list of arguments to pass to the command.
+uris (optional)          | List of resources to download into the task sandbox.
+shell (optional)         | Run executor via shell.
+
+A note on the command property (from [mesos.proto](https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto)):
+```
+1) If 'shell == true', the command will be launched via shell
+   (i.e., /bin/sh -c 'value'). The 'value' specified will be
+   treated as the shell command. The 'arguments' will be ignored.
+2) If 'shell == false', the command will be launched by passing
+   arguments to an executable. The 'value' specified will be
+   treated as the filename of the executable. The 'arguments'
+   will be treated as the arguments to the executable. This is
+   similar to how POSIX exec families launch processes (i.e.,
+   execlp(value, arguments(0), arguments(1), ...)).
+```
+
+##### uris (list)
+* Follows the [Mesos Fetcher schema](http://mesos.apache.org/documentation/latest/fetcher/)
+
+Property                 | Description
+-----------------------  | ---------------------------------
+value (required)         | Path to the resource needed in the sandbox.
+executable (optional)    | Change resource to be executable via chmod.
+extract (optional)       | Extract files from packed or compressed archives into the sandbox.
+cache (optional)         | Use caching mechanism provided by Mesos for resources.
+
+#### resources (list)
+
+Property             | Description
+-------------------  | ---------------------------------
+name (required)      | Name of the resource: cpus or mem.
+type (required)      | Type of resource. Should always be SCALAR.
+scalar (required)    | Value in float for cpus or int for mem (in MBs)
+
+### volume_mounts (list)
+
+Property                  | Description
+------------------------  | ---------------------------------
+host_path (required)      | Host path to mount inside the container.
+container_path (required) | Path inside the container where `host_path` will be mounted.
+mode (required)           | Mode in which to mount the volume, Read-Write (RW) or Read-Only
(RO).
+
+A sample configuration is as follows:
+```
+[
+    {
+      "executor": {
+        "name": "myExecutor",
+        "command": {
+          "value": "myExecutor.a",
+          "shell": "false",
+          "arguments": [
+            "localhost:2181",
+            "-verbose",
+            "-config myConfiguration.config"
+          ],
+          "uris": [
+            {
+              "value": "/dist/myExecutor.a",
+              "executable": true,
+              "extract": false,
+              "cache": true
+            },
+            {
+              "value": "/home/user/myConfiguration.config",
+              "executable": false,
+              "extract": false,
+              "cache": false
+            }
+          ]
+        },
+        "resources": [
+          {
+            "name": "cpus",
+            "type": "SCALAR",
+            "scalar": {
+              "value": 1.00
+            }
+          },
+          {
+            "name": "mem",
+            "type": "SCALAR",
+            "scalar": {
+              "value": 512
+            }
+          }
+        ]
+      },
+      "volume_mounts": [
+        {
+          "mode": "RO",
+          "container_path": "/path/on/container",
+          "host_path": "/path/to/host/directory"
+        },
+        {
+          "mode": "RW",
+          "container_path": "/container",
+          "host_path": "/host"
+        }
+      ],
+      "task_prefix": "my-executor-"
+    }
+]
+
+```
+
+It should be noted that if you do not use thermos or a thermos based executor, links in the
scheduler's
+Web UI for tasks will not work (at least for the time being).
+Some information about launched tasks can still be accessed via the Mesos Web UI or via the
Aurora Client.
+
+### Using a custom executor
+
+At this time, it is not currently possible create a job that runs on a custom executor using
the default
+Aurora client. To allow the scheduler to pick the correct executor, the `JobConfiguration.TaskConfig.ExecutorConfig.name`
+field must be set to match the name used in the custom executor configuration blob. (e.g.
to run a job using myExecutor,
+`JobConfiguration.TaskConfig.ExecutorConfig.name` must be set to `myExecutor`). While support
for modifying
+this field in Pystachio created, the easiest way to launch jobs with custom executors is
to use
+an existing custom Client such as [gorealis](https://github.com/rdelval/gorealis).
+

http://git-wip-us.apache.org/repos/asf/aurora/blob/370813f1/docs/operations/configuration.md
----------------------------------------------------------------------
diff --git a/docs/operations/configuration.md b/docs/operations/configuration.md
index bbcffac..15ec042 100644
--- a/docs/operations/configuration.md
+++ b/docs/operations/configuration.md
@@ -148,145 +148,13 @@ If the need arises to use a Mesos executor other than the Thermos executor,
the
 configured to utilize a custom executor by specifying the `-custom_executor_config` flag.
 The flag must be set to the path of a valid executor configuration file.
 
-The configuration file must be a valid **JSON array** and contain, at minimum,
-one executor configuration including the name, command and resources fields.
-
-### Array Entry
-
-Property                 | Description
------------------------  | ---------------------------------
-executor (required)      | Description of executor.
-task_prefix (required) ) | Prefix given to tasks launched with this executor's configuration.
-volume_mounts (optional) | Volumes to be mounted in container running executor.
-
-#### executor
-
-Property                 | Description
------------------------  | ---------------------------------
-name (required)          | Name of the executor.
-command (required)       | How to run the executor.
-resources (required)     | Overhead to use for each executor instance.
-
-#### command
-
-Property                 | Description
------------------------  | ---------------------------------
-value (required)         | The command to execute.
-arguments (optional)     | A list of arguments to pass to the command.
-uris (optional)          | List of resources to download into the task sandbox.
-shell (optional)         | Run executor via shell.
-
-A note on the command property (from [mesos.proto](https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto)):
-```
-1) If 'shell == true', the command will be launched via shell
-   (i.e., /bin/sh -c 'value'). The 'value' specified will be
-   treated as the shell command. The 'arguments' will be ignored.
-2) If 'shell == false', the command will be launched by passing
-   arguments to an executable. The 'value' specified will be
-   treated as the filename of the executable. The 'arguments'
-   will be treated as the arguments to the executable. This is
-   similar to how POSIX exec families launch processes (i.e.,
-   execlp(value, arguments(0), arguments(1), ...)).
-```
-
-##### uris (list)
-* Follows the [Mesos Fetcher schema](http://mesos.apache.org/documentation/latest/fetcher/)
-
-Property                 | Description
------------------------  | ---------------------------------
-value (required)         | Path to the resource needed in the sandbox.
-executable (optional)    | Change resource to be executable via chmod.
-extract (optional)       | Extract files from packed or compressed archives into the sandbox.
-cache (optional)         | Use caching mechanism provided by Mesos for resources.
-
-#### resources (list)
-
-Property             | Description
--------------------  | ---------------------------------
-name (required)      | Name of the resource: cpus or mem.
-type (required)      | Type of resource. Should always be SCALAR.
-scalar (required)    | Value in float for cpus or int for mem (in MBs)
-
-### volume_mounts (list)
-
-Property                  | Description
-------------------------  | ---------------------------------
-host_path (required)      | Host path to mount inside the container.
-container_path (required) | Path inside the container where `host_path` will be mounted.
-mode (required)           | Mode in which to mount the volume, Read-Write (RW) or Read-Only
(RO).
-
-A sample configuration is as follows:
-```
-[
-    {
-      "executor": {
-        "name": "myExecutor",
-        "command": {
-          "value": "myExecutor.a",
-          "shell": "false",
-          "arguments": [
-            "localhost:2181",
-            "-verbose",
-            "-config myConfiguration.config"
-          ],
-          "uris": [
-            {
-              "value": "/dist/myExecutor.a",
-              "executable": true,
-              "extract": false,
-              "cache": true
-            },
-            {
-              "value": "/home/user/myConfiguration.config",
-              "executable": false,
-              "extract": false,
-              "cache": false
-            }
-          ]
-        },
-        "resources": [
-          {
-            "name": "cpus",
-            "type": "SCALAR",
-            "scalar": {
-              "value": 1.00
-            }
-          },
-          {
-            "name": "mem",
-            "type": "SCALAR",
-            "scalar": {
-              "value": 512
-            }
-          }
-        ]
-      },
-      "volume_mounts": [
-        {
-          "mode": "RO",
-          "container_path": "/path/on/container",
-          "host_path": "/path/to/host/directory"
-        },
-        {
-          "mode": "RW",
-          "container_path": "/container",
-          "host_path": "/host"
-        }
-      ],
-      "task_prefix": "my-executor-"
-    }
-]
-
-```
-
-It should be noted that if you do not use thermos or a thermos based executor, links in the
scheduler's
-Web UI for tasks will not work (at least for the time being).
-Some information about launched tasks can still be accessed via the Mesos Web UI or via the
Aurora Client.
-
-### A note on increasing executor overhead
+For more information on this feature please see the custom executors [documentation](../features/custom-executors.md).
+
+## A note on increasing executor overhead
 
 Increasing executor overhead on an existing cluster, whether it be for custom executors or
for thermos,
-will result in degraded preemption performance until all task with lesser overhead are preempted/restarted.
+will result in degraded preemption performance until all task which began life with the previous
+executor configuration with less overhead are preempted/restarted.
 
 ## Docker containers
 In order for Aurora to launch jobs using docker containers, a few extra configuration options


Mime
View raw message