myriad-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dar...@apache.org
Subject incubator-myriad git commit: Release prep: -Version bump 0.2.0 -Update Docs -Update Sample Configuration -Minor changes to docker scripts Pull Request: Closes: #72
Date Thu, 12 May 2016 21:02:55 GMT
Repository: incubator-myriad
Updated Branches:
  refs/heads/master 12a679c2b -> 048727634


Release prep:
-Version bump 0.2.0
-Update Docs
-Update Sample Configuration
-Minor changes to docker scripts
Pull Request:
  Closes: #72


Project: http://git-wip-us.apache.org/repos/asf/incubator-myriad/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-myriad/commit/04872763
Tree: http://git-wip-us.apache.org/repos/asf/incubator-myriad/tree/04872763
Diff: http://git-wip-us.apache.org/repos/asf/incubator-myriad/diff/04872763

Branch: refs/heads/master
Commit: 04872763495a2c21ae2649b4c8134933cb59d932
Parents: 12a679c
Author: darinj <darinj.work@gmail.com>
Authored: Thu May 12 14:56:49 2016 -0400
Committer: darinj <darinj@apache.org>
Committed: Thu May 12 16:50:56 2016 -0400

----------------------------------------------------------------------
 README.md                                       |  3 +-
 build.gradle                                    |  2 +-
 docker/Dockerfile                               | 11 +++-
 docker/README.md                                | 60 +++++++++++++-------
 docker/myriad-bin/configure-yarn.sh             | 32 +++++------
 docker/myriad-bin/install-yarn.sh               |  4 +-
 .../myriad-config-default.yml.template          | 43 +++-----------
 docker/myriad-etc/yarn-site.xml.template        | 49 +---------------
 docs/cgroups.md                                 |  2 +-
 docs/myriad-docker.md                           |  0
 docs/myriad-multi-tenancy.md                    | 10 ++++
 .../myriad-remote-distribution-configuration.md | 24 ++++++--
 .../main/resources/myriad-config-default.yml    |  6 +-
 13 files changed, 115 insertions(+), 131 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 02925ae..bffcf79 100644
--- a/README.md
+++ b/README.md
@@ -37,7 +37,8 @@ Please keep checking this section for updates.
 - [x] Framework checkpointing
 - [x] Launch Job History Server
 - [x] Constraints based Node Manager placement
-- [ ] Support multi-tenancy for Node Managers
+- [x] Docker support
+- [x] Support multi-tenancy for Node Managers
 - [ ] Configuration store for storing rules and policies for clusters managed by Myriad
 
 ## Mailing Lists

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/build.gradle
----------------------------------------------------------------------
diff --git a/build.gradle b/build.gradle
index ac416ec..55e8d85 100644
--- a/build.gradle
+++ b/build.gradle
@@ -64,7 +64,7 @@ subprojects {
     mainClassName = "org.apache.myriad.Main"
 
     group = "org.apache.myriad"
-    version = "0.1.0"
+    version = "0.2.0"
 
     ext {
         mesosVer = "0.28.1"

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/docker/Dockerfile
----------------------------------------------------------------------
diff --git a/docker/Dockerfile b/docker/Dockerfile
index 4f344a6..8626e13 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -22,12 +22,17 @@
 FROM debian
 MAINTAINER Apache Myriad dev@myriad.incubator.apache.org
 
-ENV HADOOP_USER="yarn"
+ENV YARN_USER="yarn"
+ENV YARN_UID="108"
+ENV HADOOP_GROUP="hadoop"
+ENV HADOOP_GID="112"
+ENV YARN_GROUP="yarn"
+ENV YARN_GID="113"
 ENV HADOOP_VER="2.7.0"
 ENV HADOOP_HOME=/usr/local/hadoop
 ENV JAVA_HOME=/usr
 ENV MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
-
+ENV MESOS_VER=0.28.1
 
 # Setup mesosphere repositories
 RUN apt-get -y update
@@ -37,7 +42,7 @@ RUN DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') CODENAME=$(lsb_releas
 RUN apt-get -y update
 
 # Install Mesos
-RUN apt-get install -y mesos curl tar
+RUN apt-get install -y mesos=`apt-cache madison mesos | grep $MESOS_VER | head -1 | awk {'print
$3'}` curl tar
 # Run local YARN install
 ADD myriad-bin/install-yarn.sh /install-yarn.sh
 RUN sh /install-yarn.sh

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/docker/README.md
----------------------------------------------------------------------
diff --git a/docker/README.md b/docker/README.md
index 1650691..f8fdb62 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -1,41 +1,63 @@
 # Using Myriad with Docker #
 Docker is the easiest way to from 0 to Yarn on Mesos within minutes. 
 
-## Building the Resource Manager Docker
+# Building the Resource Manager Docker
 
-#Configuration Guide#
+YARN runs as a user and group, and expects consistent uid and gids in HDFS and accross the
cluster.  Hence it is necessary 
+to edit docker/Dockerfile and modify the following:
+```
+ENV YARN_USER="yarn"
+ENV YARN_UID="107"
+ENV HADOOP_GROUP="hadoop"
+ENV HADOOP_GID="112"
+ENV YARN_GROUP="yarn"
+ENV YARN_UID="113"
+```
+
+The run the following commands:
+```bash
+./gradlew -P dockerTag=username/myriad buildRMDocker
+docker push username/myriad
+```
+
+This will build the ResourceManager from src, save, and push the image as *username/myriad*.
+
+# Configuration Guide
+
+It is assumed you already have hdfs and Mesos running.  For more information about Apache
Mesos visit the [website](http://mesos.apache.org). 
+If you need to setup hdfs, consider using the [hdfs-mesos framework](https://github.com/mesosphere/hdfs).
 
 In order for the ResourceManager to operate correctly, you will need to provide 5 configuration
files. These files will 
-need to mounted from a directory into `./myriad-etc` of this folder.
+need to mounted from a directory into `/usr/local/hadoo/etc` of the docker container.
 
-* [myriad-config-default.yml](https://github.com/mesos/myriad/blob/phase1/myriad-scheduler/src/main/resources/myriad-config-default.yml)
(template provided)
-* [yarn-site.xml](https://github.com/mesos/myriad/blob/phase1/docs/myriad-dev.md) (template
provided)
+* [myriad-config-default.yml](https://github.com/apacher/incubator-myriad/blob/phase1/myriad-scheduler/src/main/resources/myriad-config-default.yml)
(template provided)
+* [yarn-site.xml](https://github.com/apache/incubator-myriad/blob/phase1/docs/myriad-dev.md)
(template provided)
 * mapred-site.xml (template provided)
 * hdfs-site.xml (used for hdfs)
 * core-site.xml (used for hdfs)
 
-It is assumed you already have hdfs and Mesos running.  For more information about Apache
Mesos visit the [website](http://mesos.apache.org). 
-If you need to setup hdfs, consider using the [hdfs-mesos framework](https://github.com/mesosphere/hdfs).
 Copy the hdfs.xml and 
-core-site.xml from your hdfs configuration into myriad-etc.  Additional, files maybe necessary
such as rack.sh, log4j.properties, 
-hadoop-env.sh, and yarn-env.sh depending on your configuration.
+Edit and copy the file templates along with the hdfs.xml and core-site.xml from your hdfs
configuration into a directory named `config`.  Additional, files maybe necessary such as
rack.sh, log4j.properties, 
+hadoop-env.sh, and yarn-env.sh depending on your configuration. Create a second directoy
called `dist`. Then tar the files using `tar -zcf dist\config.tgz config`.
 
-Run the following commands:
+# Running the Resource Manager Docker
+
+From the directories containing `dist` and `config` execute:
 ```bash
-./gradlew -P username/myriad buildRMDocker
-docker push username/myriad
+docker run --net=host -v $PWD/dist -v $PWD/config:/usr/local/hadoop/etc/hadoop --name='myriad-resourcemanager'
-t \
+  <username>/myriad
 ```
-This will build the ResourceManager from src, save, and push the image as *username/myriad*.
+# Updating configuration
 
-## Running the Resource Manager Docker
+There is no need to to rebuild the docker image to update YARN's configuration, simply edit
the files in config and repackage using 
+`tar -zcf dist\config.tgz config`.  To update the configuration on the Nodemanagers they
will need to be flexed down and flexed up, to 
+ update the resource manager the docker container will need to be restarted:
 
 ```bash
-docker run --net=host --name='myriad-resourcemanager' -t \
+docker kill myriad-resourcemanager
+docker run --net=host -v $PWD/dist -v $PWD/config:/usr/local/hadoop/etc/hadoop --name='myriad-resourcemanager'
-t \
   <username>/myriad
 ```
-
-#Environment Variables#
-* *ALTERNATE_HADOOP_URL* : Optional - Allows user to override the hadoop distribution used
by Myriad. This will download the *.tar.gz file to be used as the hadoop distribution of choice
for Myriad. 
-
+ 
 ---
 <sub>
 Licensed to the Apache Software Foundation (ASF) under one

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/docker/myriad-bin/configure-yarn.sh
----------------------------------------------------------------------
diff --git a/docker/myriad-bin/configure-yarn.sh b/docker/myriad-bin/configure-yarn.sh
index ff85ba9..7ba8f45 100755
--- a/docker/myriad-bin/configure-yarn.sh
+++ b/docker/myriad-bin/configure-yarn.sh
@@ -22,20 +22,20 @@ under the License.
 ##
 
 # Put in env defaults if they are missing
-export HADOOP_GROUP=${HADOOP_GROUP:='hadoop'}
-
 export YARN_USER=${YARN_USER:='yarn'}
-export USER_UID=${USER_UID:='107'}
+export YARN_UID=${YARN_UID:='107'}
 export YARN_GROUP=${YARN_GROUP:='yarn'}
+export HADOOP_GROUP=${HADOOP_GROUP:='hadoop'}
 export HADOOP_GID=${HADOOP_GID:='112'}
 export YARN_GID=${YARN_GID:='113'}
 export HADOOP_HOME=${HADOOP_HOME:='/usr/local/hadoop'}
+export MESOS_NATIVE_LIBRARY=${MESOS_NATIVE_LIBRARY:='/usr/local/lib/libmesos.so'}
 
 # Add hduser user
 groupadd ${HADOOP_GROUP} -g ${HADOOP_GID}
 groupadd ${YARN_GROUP} -g ${YARN_GID}
-useradd ${YARN_USER} -g ${YARN_GROUP} -G ${HADOOP_GROUP} -u ${USER_UID} -s /bin/bash
-mkdir /home/${HADOOP_USER}
+useradd ${YARN_USER} -g ${YARN_GROUP} -G ${HADOOP_GROUP} -u ${YARN_UID} -s /bin/bash
+mkdir /home/${YARN_USER}
 chown -R ${YARN_USER}:${YARN_GROUP} /home/${YARN_USER}
 
 #set permissions
@@ -45,16 +45,16 @@ chown root:${YARN_GROUP} ${HADOOP_HOME}/bin/container-executor
 chmod 6050 ${HADOOP_HOME}/bin/container-executor
 
 # Init bashrc with hadoop env variables
-sh -c 'echo export JAVA_HOME=/usr >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export PATH=\$PATH:\${HADOOP_HOME}/bin >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export PATH=\$PATH:\${HADOOP_HOME}/sbin >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_MAPRED_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_COMMON_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_HDFS_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export YARN_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_COMMON_LIB_NATIVE_DIR=\$\{HADOOP_HOME\}/lib/native >> /home/${HADOOP_USER}/.bashrc'
-sh -c 'echo export HADOOP_OPTS=\"-Djava.library.path=\${HADOOP_HOME}/lib\" >> /home/${HADOOP_USER}/.bashrc'
-
+sh -c 'echo export JAVA_HOME=/usr >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export MESOS_NATIVE_LIBRARY=/usr >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_HOME=\${HADOOP_HOME} >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export PATH=\$PATH:\${HADOOP_HOME}/bin >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export PATH=\$PATH:\${HADOOP_HOME}/sbin >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_MAPRED_HOME=\${HADOOP_HOME} >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_COMMON_HOME=\${HADOOP_HOME} >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_HDFS_HOME=\${HADOOP_HOME} >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export YARN_HOME=\${HADOOP_HOME} >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_COMMON_LIB_NATIVE_DIR=\$\{HADOOP_HOME\}/lib/native >> /home/${YARN_USER}/.bashrc'
+sh -c 'echo export HADOOP_OPTS=\"-Djava.library.path=\${HADOOP_HOME}/lib\" >> /home/${YARN_USER}/.bashrc'
 
 echo "end of configure-yarn.sh script"

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/docker/myriad-bin/install-yarn.sh
----------------------------------------------------------------------
diff --git a/docker/myriad-bin/install-yarn.sh b/docker/myriad-bin/install-yarn.sh
index 90c2762..fb8453e 100755
--- a/docker/myriad-bin/install-yarn.sh
+++ b/docker/myriad-bin/install-yarn.sh
@@ -20,7 +20,8 @@ under the License.
 ##
 # YARN installer script for Apache Myriad Deployment
 ##
-HADOOP_VER="2.7.0"
+export HADOOP_VER=${HADOOP_VER:='2.7.0'}
+export HADOOP_HOME=${HADOOP_HOME:='/usr/local/hadoop'}
 
 echo "Installing Yarn...."
 if [ ! -z $1 ];then
@@ -30,7 +31,6 @@ else
 fi
 HADOOP_TGZ=`basename ${HADOOP_URL}`
 HADOOP_BASENAME=`basename ${HADOOP_URL} .tar.gz`
-export HADOOP_HOME=${HADOOP_HOME:='/usr/local/hadoop'}
 
 # Extract Hadoop
 echo "Downloading ${HADOOP_TGZ} from ${HADOOP_URL}"

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/docker/myriad-etc/myriad-config-default.yml.template
----------------------------------------------------------------------
diff --git a/docker/myriad-etc/myriad-config-default.yml.template b/docker/myriad-etc/myriad-config-default.yml.template
index 6a1e52a..bfbc805 100644
--- a/docker/myriad-etc/myriad-config-default.yml.template
+++ b/docker/myriad-etc/myriad-config-default.yml.template
@@ -18,12 +18,12 @@
 ##
 
 # Address of the mesos master - <IP:port> or ZooKeeper path
-mesosMaster: zk://CHANGEME:2181/mesos
+mesosMaster: zk://<ZOOKEEPER>:2181/mesos
 #Container information for the node managers
 containerInfo:
     type: DOCKER
     dockerInfo:
-        image: CHANGEME/myriad
+        image: <IMAGE>/myriad
     volume:
         -
           containerPath: /tmp
@@ -35,19 +35,13 @@ checkpoint: false
 frameworkFailoverTimeout: 60000 #10 minutes
 # Myriad's mesos framework name.
 frameworkName: MyriadAlpha
-# Myriad's mesos framework role.
-#frameworkRole: myriad
 # User the Node Manager will run as (Defaults to user running the resource manager if absent,
 necessary for remote distribution).
 frameworkUser: yarn
-# User that gets the nodeManagerUri and sets up the directories for Node Manager, must have
passwordless sudo (Necessary only for remote distribution, otherwise ignored).
 # frameworkSuperUser: root
 # Myriad's REST-ful services port mapping.
 restApiPort: 8192
-# Address of the ZK ensemble (separate by comma, if multiple zk servers are used)
-#zkServers: localhost:2181
-# ZK Session timeout
-zkTimeout: 20000
 haEnabled: True
+servedConfigPath: /dist/config.tgz
 # The node manager profiles. The REST API to flex up expects one of the profiles defined
here.
 # Admin can define custom profiles (requires restart of Resource Manager)
 nmInstances:
@@ -75,37 +69,18 @@ rebalancer: false
 nodemanager:
   jvmMaxMemoryMB: 1024  # Xmx for NM JVM process.
   cpus: 0.2             # CPU needed by NM process.
-#  cgroups: true         # Whether NM should support CGroups. If set to 'true', myriad automatically
-                        # configures yarn-site.xml to attach YARN's cgroups under Mesos'
cgroup hierarchy.
+
 executor:
   jvmMaxMemoryMB: 256   # Xmx for myriad's executor that launches Node Manager.
-  #path: hdfs://172.31.2.176:54310/dist/myriad-executor-runnable-x.x.x.jar # Path for the
myriad's executor binary.
-  nodeManagerUri: hdfs://172.31.2.167:54310/dist/hadoop-2.6.0.tgz # the uri to d/l hadoop
from   # Path to the Hadoop tarball
+  configUri: http://<RESOURCEMANAGER>:8192/api/config.tgz
 # Environment variables required to launch Node Manager process. Admin can also pass other
environment variables to NodeManager.
 yarnEnvironment:
   YARN_HOME: /usr/local/hadoop # Or /usr/local/hadoop if using MapR's Hadoop
-  #YARN_HOME: hadoop-2.6.0 # Should be relative nodeManagerUri is set
-  #YARN_NODEMANAGER_OPTS: -Dnodemanager.resource.io-spindles=4.0 # Required only if using
MapR's Hadoop
   JAVA_HOME: /usr/ # System dependent, but sometimes necessary
-services:
-   jobhistory:
-     command: $YARN_HOME/bin/mapred historyserver
-     jvmMaxMemoryMB: 1024
-     cpus: 1
-     maxInstances: 1
-     #ports:
-      # myriad.mapreduce.jobhistory.admin.address: -1
-      # myriad.mapreduce.jobhistory.address: -1
-      # myriad.mapreduce.jobhistory.webapp.address: -1
-     envSettings: -Dcluster.name.prefix=/mycluster
-     taskName: jobhistory
-  timelineserver:
-     command: $YARN_HOME/bin/yarn timelineserver
-     jvmMaxMemoryMB: 1024
-     cpus: 1
-     envSettings: -Dcluster.name.prefix=/mycluster2
-     taskName: timelineserver
-
+  HADOOP_CONF_DIR: /mnt/mesos/sandbox/config
+  HADOOP_TMP_DIR: /tmp
+  HADOOP_LOG_DIR: /mnt/mesos/sandbox
+  #YARN_NODEMANAGER_OPTS: -Dnodemanager.resource.io-spindles=4.0 # Required only if using
MapR's Hadoop
 
 # Authentication principal for Myriad's mesos framework
 #mesosAuthenticationPrincipal: some_principal

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/docker/myriad-etc/yarn-site.xml.template
----------------------------------------------------------------------
diff --git a/docker/myriad-etc/yarn-site.xml.template b/docker/myriad-etc/yarn-site.xml.template
index c81b124..fc14726 100644
--- a/docker/myriad-etc/yarn-site.xml.template
+++ b/docker/myriad-etc/yarn-site.xml.template
@@ -17,8 +17,6 @@
 -->
 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
-
-
 <configuration>
 
   <!--The hostname of the Resource Manager, this MUST be changed! -->
@@ -76,6 +74,7 @@
     <name>yarn.nodemanager.resource.cpu-vcores</name>
     <value>${nodemanager.resource.cpu-vcores}</value>
    </property>
+
    <property>
      <name>yarn.scheduler.minimum-allocation-vcores</name>
      <value>0</value>
@@ -85,6 +84,7 @@
     <name>yarn.nodemanager.resource.memory-mb</name>
     <value>${nodemanager.resource.memory-mb}</value>
   </property>
+
   <property>
     <name>yarn.scheduler.minimum-allocation-mb</name>
     <value>0</value>
@@ -114,49 +114,4 @@
     <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value>
   </property>
 
-  <!-- Cgroups configuration -->
-  <!--
-  <property>
-    <description>who will execute(launch) the containers.</description>
-    <name>yarn.nodemanager.container-executor.class</name>
-    <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
-  </property>
-
-  <property>
-    <description>The class which should help the LCE handle resources.</description>
-    <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
-    <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value>
-  </property>
-
-  <property>
-    <description>The class which should help the LCE handle resources.</description>
-    <name>yarn.nodemanager.linux-container-executor.group</name>
-    <value>yarn</value>
-  </property>
-
-  <property>
-    <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
-    <value></value>
-  </property>
-
-  <property>
-    <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
-    <value>true</value>
-  </property>
-
-  <property>
-    <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
-    <value>${yarn.nodemanager.linux-container-executor.cgroups.mount-path}</value>
-  </property>
-
-  <property>
-    <name>yarn.nodemanager.linux-container-executor.group</name>
-    <value>root</value>
-  </property>
-
-  <property>
-    <name>yarn.nodemanager.linux-container-executor.path</name>
-    <value>${yarn.home}/bin/container-executor</value>
-  </property>
--->
 </configuration>

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/docs/cgroups.md
----------------------------------------------------------------------
diff --git a/docs/cgroups.md b/docs/cgroups.md
index 8694db8..d28eeab 100644
--- a/docs/cgroups.md
+++ b/docs/cgroups.md
@@ -1,5 +1,5 @@
 # Configuring Cgroups
-
+** CGroups are currently not supported when using Docker, we are aware this feature may be
of interest and will add soon ** 
 A node within a cluster is the location where tasks are run. The tasks are launched with
help of a daemon which resides inside the node. This daemon, in the case of Mesos, is Mesos
Slave, and in case of YARN, is NodeManager.
 
 The Cgroups Linux kernel feature allows aggregating or partitioning a set of tasks and their
future children into hierarchical groups with respect to one or more subsystems. For example,
when Cgroups is enabled for the CPU subsystem, and a task is launched by Mesos Slave, it goes
under the following hierarchy:

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/docs/myriad-docker.md
----------------------------------------------------------------------
diff --git a/docs/myriad-docker.md b/docs/myriad-docker.md
new file mode 100644
index 0000000..e69de29

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/docs/myriad-multi-tenancy.md
----------------------------------------------------------------------
diff --git a/docs/myriad-multi-tenancy.md b/docs/myriad-multi-tenancy.md
new file mode 100644
index 0000000..3eaa4f0
--- /dev/null
+++ b/docs/myriad-multi-tenancy.md
@@ -0,0 +1,10 @@
+# Myriad Multi-tenacy
+
+Myriad supports multi-tenancy in the following ways:
+- Myriad uses Mesos to assign ports for the Node Managers.  This means if to Node Managers
run on the same host, they will both be able to 
+bind to a port and communicate
+- Myriad provides mechanisms to distribute the hadoop binaries and configurations using [tarballs](https://github.com/apache/incubator-myriad/blob/master/docs/myriad-remote-distribution-configuration.md)
or [Docker](https://github.com/apache/incubator-myriad/blob/master/docker/README.md).
+
+This means it's entirely possible to run multiple Myriad Clusters on the same hardware. One
potential issue to be aware of however is it 
+is often important when configuring yarn to use multiple directories.  If one needs to do
this one should be careful and not use the directories 
+for to different Myriad frameworks.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/docs/myriad-remote-distribution-configuration.md
----------------------------------------------------------------------
diff --git a/docs/myriad-remote-distribution-configuration.md b/docs/myriad-remote-distribution-configuration.md
index 4132b76..2eae88e 100644
--- a/docs/myriad-remote-distribution-configuration.md
+++ b/docs/myriad-remote-distribution-configuration.md
@@ -18,8 +18,6 @@ The following are assumptions about your environment:
 
 * You are using hadoop-2.7.0 downloaded from [hadoop.apache.org](http://hadoop.apache.org).
 Specific vendor versions should work but may require additional steps.
 
-**NOTE:** The default location for $YARN_HOME is **/opt/hadoop-2.7.0**.
-
 ## Building the Myriad Remote Distribution Bundle ##
 Before building Myriad, configure the Resource Manager as you normally would.
 
@@ -49,14 +47,19 @@ Edit the **$YARN_HOME/etc/hadoop/myriad-config-default.yml** file to configure
t
 frameworkSuperUser: admin              # Must be root or have passwordless sudo on all nodes!
 frameworkUser: hduser                  # Should be the same user running the resource manager.
                                        # Must exist on all nodes and be in the 'hadoop' group
+#servedConfigPath: /dist/config.tgz
+#servedBinaryPath: /dist/hadoop-2.7.0.tar.gz
 executor:
   nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.7.0.tar.gz
+  #nodeManagerUri: http://resourcemanager/api/binary.tgz
+  #configUri: http://resourcemanager/api/config.tgz
 yarnEnvironment:
-YARN_HOME: hadoop-2.7.0                # This should be relative if nodeManagerUri is set
+  YARN_HOME: hadoop-2.7.0                # This should be relative if nodeManagerUri is set
+  JAVA_HOME: /usr/
+  #HADOOP_CONF_DIR: config
 ```
 
 
-
 ### Step 4: Configure YARN to Use Myriad ###
 
 Modify the  **YARN_HOME/etc/hadoop/yarn-site.xml** file. See [Myriad yarn-site.xml file](sample-yarn-site.md)
@@ -74,6 +77,19 @@ sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
 hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
 ```
 
+Alternatively, one can serve the tar ball directly from Myriad using the `servedBinaryPath`
parameter (see the commented line in the code block of step 3).
+
+### Advanced Configuration ###
+
+By default Myriad has the Node Managers pull the yarn-site.xml from the resource managers
configuration url.  While this is convenient,
+it is sometimes preferable to package and distribute the hadoop configuration files separately.
 Myriad provides these facilities using 
+`servedConfigPath` and `configUri` options and using `yarnEnvironment` to pass `HADOOP_CONF_DIR`
(see the commented lines in the code block of step 3).
+
+By default Myriad assumes that Java is preinstalled on each Mesos slave.  While this is convenient,
+it is sometimes preferable to package and distribute the JDK seperately.  Myriad provides
this facility using 
+`jvmUri` option and using `yarnEnvironment` to pass `JAVA_HOME` (see the commented lines
in the code block of step 3).
+
+
 ## Getting Started ##
 
 You can now start the resource manager and attempt to flexup or flexdown the cluster. See
 [Getting Started](getting-started.md) for information about using Myriad. See the [Myriad
Cluster API](API.md) for more information about scaling.

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/04872763/myriad-scheduler/src/main/resources/myriad-config-default.yml
----------------------------------------------------------------------
diff --git a/myriad-scheduler/src/main/resources/myriad-config-default.yml b/myriad-scheduler/src/main/resources/myriad-config-default.yml
index a2c143d..f985054 100644
--- a/myriad-scheduler/src/main/resources/myriad-config-default.yml
+++ b/myriad-scheduler/src/main/resources/myriad-config-default.yml
@@ -58,14 +58,14 @@ executor:
   #The following should be used for a remotely distributed URI, hdfs assumed but other URI
types valid.
   #nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.7.0.tar.gz
   #configUri: http://127.0.0.1/api/arifacts/config.tgz
-  #path: hdfs://namenode:port/dist/myriad-executor-runnable-0.1.0.jar
   #jvmUri: https://downloads.mycompany.com/java/jre-7u76-linux-x64.tar.gz
 yarnEnvironment:
   YARN_HOME: /usr/local/hadoop
   #HADOOP_CONF_DIR=config
+  #HADOOP_TMP_DIR=$MESOS_SANDBOX
   #YARN_HOME: hadoop-2.7.0 #this should be relative if nodeManagerUri is set
   #JAVA_HOME: /usr/lib/jvm/java-default #System dependent, but sometimes necessary
   #JAVA_HOME: jre1.7.0_76 # Path to JRE distribution, relative to sandbox directory
   #JAVA_LIBRARY_PATH: /opt/mycompany/lib
-mesosAuthenticationPrincipal:
-mesosAuthenticationSecretFilename:
+#mesosAuthenticationPrincipal:
+#mesosAuthenticationSecretFilename:


Mime
View raw message