myriad-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From smare...@apache.org
Subject [2/3] incubator-myriad git commit: Refactored ResourceManager Docker support
Date Tue, 20 Oct 2015 18:34:42 GMT
Refactored ResourceManager Docker support

Highlights:
  - Hadoop distribution is now user definable. Use the HADOOP_TARBALL_URL environment variable
to force the docker to download and use your distribution.
  - Hadoop 2.7.1 is now the default distribution used.
  - The base image is now based off of Debian for leaner docker images.
  - ResourceManager now runs as 'yarn' user.

Addresses MYRIAD-146

This closes: #13
Review: https://github.com/apache/incubator-myriad/pull/13


Project: http://git-wip-us.apache.org/repos/asf/incubator-myriad/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-myriad/commit/71808ea6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-myriad/tree/71808ea6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-myriad/diff/71808ea6

Branch: refs/heads/master
Commit: 71808ea6c8be198b9824f206826c76a8e61fe115
Parents: 5cab3b1
Author: root <root@ec2-54-72-62-8.eu-west-1.compute.amazonaws.com>
Authored: Tue Oct 20 11:29:54 2015 -0700
Committer: Santosh Marella <marella@gmail.com>
Committed: Tue Oct 20 11:33:09 2015 -0700

----------------------------------------------------------------------
 docker/Dockerfile                 | 44 +++++++++++++++++++++++++
 docker/README.md                  | 26 +++++++--------
 docker/ResourceManager.dockerfile | 24 ++++++++------
 docker/docker-compose.yml         |  9 ++++++
 docker/myriad-bin/create-user.sh  | 34 ++++++++++++++++++++
 docker/myriad-bin/install-yarn.sh | 59 +++++++++++++++++++++++++++-------
 docker/myriad-bin/run-myriad.sh   | 34 +++++++++++++++++---
 7 files changed, 192 insertions(+), 38 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/71808ea6/docker/Dockerfile
----------------------------------------------------------------------
diff --git a/docker/Dockerfile b/docker/Dockerfile
new file mode 100644
index 0000000..c32243c
--- /dev/null
+++ b/docker/Dockerfile
@@ -0,0 +1,44 @@
+# Myriad - YARN Resource Manager
+#
+# VERSION 0.0.1
+
+FROM debian
+MAINTAINER Apache Myriad dev@myriad.incubator.apache.org
+#USER yarn
+
+ENV HADOOP_USER="yarn"
+ENV HADOOP_VER="2.7.1"
+ENV HADOOP_TARBALL_URL="http://172.31.1.11/hadoop-2.7.1.tar.gz"
+
+#RUN useradd yarn -g 112 -u 113 -s /bin/bash
+
+# Setup mesosphere repositories
+RUN apt-get -y update
+RUN apt-get install -y openjdk-7-jre-headless wget lsb-release
+RUN apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF
+RUN DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') CODENAME=$(lsb_release -cs) &&
echo "deb http://repos.mesosphere.com/${DISTRO} ${CODENAME} main" | tee /etc/apt/sources.list.d/mesosphere.list
+RUN apt-get -y update
+
+ADD myriad-bin/create-user.sh /create-user.sh
+RUN sh /create-user.sh
+
+
+# Install Mesos
+RUN apt-get install -y mesos curl tar
+
+# Run local YARN install
+ADD myriad-bin/install-yarn.sh /install-yarn.sh
+RUN sh /install-yarn.sh
+
+# Copy over myriad libraries
+ADD /libs/myriad-executor-runnable-0.0.1.jar /usr/local/libexec/mesos/
+ADD /libs/* /usr/local/hadoop/share/hadoop/yarn/lib/
+
+
+# Initialize hadoop confs with env vars
+ADD myriad-bin/run-myriad.sh /run-myriad.sh
+RUN mkdir /myriad-conf/
+
+# Run the YARN resource manager
+USER yarn
+CMD sh /run-myriad.sh 

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/71808ea6/docker/README.md
----------------------------------------------------------------------
diff --git a/docker/README.md b/docker/README.md
index 58d993d..a2ed27e 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -4,28 +4,28 @@
 
 `./build-myriad.sh` will run the gradle scripts from the root myriad folder and compile all
necessary libraries.
 
+#Configuration Guide#
+
+In order for the ResourceManager to operate correctly, you will need to provide 2 configuration
files:
+
+* [myriad-config-default.yml](https://github.com/mesos/myriad/blob/phase1/myriad-scheduler/src/main/resources/myriad-config-default.yml)
+* modified [yarn-site.xml](https://github.com/mesos/myriad/blob/phase1/docs/myriad-dev.md)
+
+
 ## Running the Resource Manager Docker
 
 ```bash
-docker run --name='myriad-resourcemanager' -t \
-  -v /path/to/yarn-site.xml:/myriad-conf/yarn-site.xml \
-  -v /path/to/myriad-config-default.yml:/myriad-conf/myriad-config-default.yml \
+docker run --net=host --name='myriad-resourcemanager' -t \
+  -v /path/to/configs:/myriad-conf \
   -e HADOOP_NAMENODE="10.100.3.237:9000" \
   mesos/myriad-resourcemanager
   ```
 
-### Available Environment Variables
-You can also pass in custom values via docker run for the following env vars: 
-* HADOOP_HOME
-* HADOOP_USER
-* HADOOP_GROUP
-
-#Configuration Guide#
+#Environment Variables#
 
-In order for the ResourceManager to operate correctly, you will need to provide 2 configuration
files:
+* *HADOOP_NAMENODE* : *Required*
+* *ALTERNATE_HADOOP_URL* : Optional - Allows user to override the hadoop distribution used
by Myriad.
 
-* [myriad-config-default.yml](https://github.com/mesos/myriad/blob/phase1/myriad-scheduler/src/main/resources/myriad-config-default.yml)
-* modified [yarn-site.xml](https://github.com/mesos/myriad/blob/phase1/docs/myriad-dev.md)
 
 If you already had a working Vagrant instance, you will need to run `vagrant reload` in order
to allow zookeeper and hdfs port-forwarding.
 

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/71808ea6/docker/ResourceManager.dockerfile
----------------------------------------------------------------------
diff --git a/docker/ResourceManager.dockerfile b/docker/ResourceManager.dockerfile
index 8bb1d5a..064ba03 100644
--- a/docker/ResourceManager.dockerfile
+++ b/docker/ResourceManager.dockerfile
@@ -2,19 +2,25 @@
 #
 # VERSION 0.0.1
 
-FROM centos
+FROM debian
 MAINTAINER Apache Myriad dev@myriad.incubator.apache.org
 
-ENV HADOOP_VER="2.5.2"
+ENV HADOOP_USER="yarn"
+ENV HADOOP_VER="2.7.1"
 
-# Install Hadoop & Dependencies
-RUN yum -y update && yum install -y java-1.7.0-openjdk wget
-RUN wget http://apache.osuosl.org/hadoop/common/hadoop-${HADOOP_VER}/hadoop-${HADOOP_VER}.tar.gz
-RUN yum install -y tar
+# Setup mesosphere repositories
+RUN apt-get -y update
+RUN apt-get install -y openjdk-7-jre-headless wget lsb-release
+RUN apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF
+RUN DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') CODENAME=$(lsb_release -cs) &&
echo "deb http://repos.mesosphere.com/${DISTRO} ${CODENAME} main" | tee /etc/apt/sources.list.d/mesosphere.list
+RUN apt-get -y update
+
+# Create the yarn user
+ADD myriad-bin/create-user.sh /create-user.sh
+RUN sh /create-user.sh
 
 # Install Mesos
-RUN rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
-RUN yum -y install mesos
+RUN apt-get install -y mesos curl tar
 
 # Run local YARN install
 ADD myriad-bin/install-yarn.sh /install-yarn.sh
@@ -24,10 +30,10 @@ RUN sh /install-yarn.sh
 ADD /libs/myriad-executor-runnable-0.0.1.jar /usr/local/libexec/mesos/
 ADD /libs/* /usr/local/hadoop/share/hadoop/yarn/lib/
 
-
 # Initialize hadoop confs with env vars
 ADD myriad-bin/run-myriad.sh /run-myriad.sh
 RUN mkdir /myriad-conf/
 
 # Run the YARN resource manager
+USER yarn
 CMD sh /run-myriad.sh 

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/71808ea6/docker/docker-compose.yml
----------------------------------------------------------------------
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
new file mode 100644
index 0000000..407f3f3
--- /dev/null
+++ b/docker/docker-compose.yml
@@ -0,0 +1,9 @@
+myriad:
+  build: .
+  dockerfile: ResourceManager.dockerfile
+  volumes:
+    - ./config:/myriad-conf  
+  environment:
+    - "HADOOP_NAMENODE=172.31.45.229:54310"
+#    - "ALTERNATE_HADOOP_URL=http://192.168.99.100/files/hadoop-2.7.0.tar.gz"
+  net: "host"

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/71808ea6/docker/myriad-bin/create-user.sh
----------------------------------------------------------------------
diff --git a/docker/myriad-bin/create-user.sh b/docker/myriad-bin/create-user.sh
new file mode 100755
index 0000000..9d6b2c2
--- /dev/null
+++ b/docker/myriad-bin/create-user.sh
@@ -0,0 +1,34 @@
+#!/bin/bash
+: '
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+'
+# Put in env defaults if they are missing
+export HADOOP_GROUP=${HADOOP_GROUP:='hadoop'}
+export HADOOP_USER=${HADOOP_USER:='yarn'}
+export HADOOP_HOME=${HADOOP_HOME:='/usr/local/hadoop'}
+export USER_UID=${USER_UID:='113'}
+export GROUP_UID=${GROUP_GID:='112'}
+
+# Add hduser user
+echo "Creating $HADOOP_USER user.."
+groupadd $HADOOP_GROUP -g ${GROUP_UID}
+useradd $HADOOP_USER -g $HADOOP_GROUP -u ${USER_UID} -s /bin/bash
+mkdir /home/${HADOOP_USER}
+chown -R $HADOOP_USER:$HADOOP_GROUP /home/${HADOOP_USER}
+
+echo "end of create-user.sh script"

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/71808ea6/docker/myriad-bin/install-yarn.sh
----------------------------------------------------------------------
diff --git a/docker/myriad-bin/install-yarn.sh b/docker/myriad-bin/install-yarn.sh
index 4acb3c6..cd80090 100755
--- a/docker/myriad-bin/install-yarn.sh
+++ b/docker/myriad-bin/install-yarn.sh
@@ -1,27 +1,60 @@
 #!/bin/bash
+: '
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
 
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+'
+##
 # YARN installer script for Apache Myriad Deployment
-# VERSION 0.0.1
+##
+HADOOP_VER="2.7.1"
+HADOOP_TARBALL_URL=http://apache.osuosl.org/hadoop/common/hadoop-${HADOOP_VER}/hadoop-${HADOOP_VER}.tar.gz
 
-yum install -y tar curl
+echo "Installing Yarn...."
+if [ ! -z "$1" ];then
+  HADOOP_TARBALL_URL=$1
+  echo "Deleting previous hadoop home"
+  rm -rf ${HADOOP_HOME}
+fi
 
-HADOOP_VER=2.5.2
+# Download the tarball
+wget -O /opt/hadoop.tgz ${HADOOP_TARBALL_URL}
+HADOOP_BASENAME=`basename ${HADOOP_TARBALL_URL} .tar.gz`
 
 # Put in env defaults if they are missing
 export HADOOP_GROUP=${HADOOP_GROUP:='hadoop'}
-export HADOOP_USER=${HADOOP_USER:='hduser'}
+export HADOOP_USER=${HADOOP_USER:='yarn'}
 export HADOOP_HOME=${HADOOP_HOME:='/usr/local/hadoop'}
-
+export USER_UID=${USER_UID:='113'}
+export GROUP_UID=${GROUP_GID:='112'}
 
 # Add hduser user
-groupadd $HADOOP_GROUP
-useradd $HADOOP_USER -g $HADOOP_GROUP
-mkdir /home/${HADOOP_USER}
+groupadd $HADOOP_GROUP -g ${GROUP_UID}
+useradd $HADOOP_USER -g $HADOOP_GROUP -u ${USER_UID} -s /bin/bash
+#mkdir /home/${HADOOP_USER}
 chown -R $HADOOP_USER:$HADOOP_GROUP /home/${HADOOP_USER}
 
 # Extract Hadoop
-tar vxzf hadoop-${HADOOP_VER}.tar.gz -C /tmp
-mv /tmp/hadoop-${HADOOP_VER} ${HADOOP_HOME}
+tar vxzf /opt/hadoop.tgz -C /tmp
+#mv /tmp/hadoop-${HADOOP_VER} ${HADOOP_HOME}
+echo "Moving /tmp/hadoop-${HADOOP_BASENAME} to ${HADOOP_HOME}"
+mv /tmp/${HADOOP_BASENAME} ${HADOOP_HOME}
+ls -lath ${HADOOP_HOME}
+
+mkdir /home/$HADOOP_USER
 chown -R ${HADOOP_USER}:${HADOOP_GROUP} ${HADOOP_HOME}
 
 # Init bashrc with hadoop env variables
@@ -36,8 +69,12 @@ sh -c 'echo export YARN_HOME=\${HADOOP_HOME} >> /home/${HADOOP_USER}/.bashrc'
 sh -c 'echo export HADOOP_COMMON_LIB_NATIVE_DIR=\$\{HADOOP_HOME\}/lib/native >> /home/${HADOOP_USER}/.bashrc'
 sh -c 'echo export HADOOP_OPTS=\"-Djava.library.path=\${HADOOP_HOME}/lib\" >> /home/${HADOOP_USER}/.bashrc'
 
-
 # Link Mesos Libraries
+touch ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
+echo "export JAVA_HOME=/usr" >> ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
 echo "export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so" >> ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
 
+# Ensure the hadoop-env is executable
+chmod +x ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
+
 echo "end of install-yarn.sh script"

http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/71808ea6/docker/myriad-bin/run-myriad.sh
----------------------------------------------------------------------
diff --git a/docker/myriad-bin/run-myriad.sh b/docker/myriad-bin/run-myriad.sh
index 9828fef..5783f48 100755
--- a/docker/myriad-bin/run-myriad.sh
+++ b/docker/myriad-bin/run-myriad.sh
@@ -1,5 +1,31 @@
 #!/bin/bash
-
+: '
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+'
+
+
+echo "Starting Myriad..."
+
+if [ ! -z $ALTERNATE_HADOOP_URL ]; then
+  echo "OVERIDING URL"
+  exec /install-yarn.sh $ALTERNATE_HADOOP_URL
+  unset $ALTERNATE_HADOOP_URL
+fi
 
 export MYRIAD_CONFIG_FILE="/myriad-conf/myriad-config-default.yml"
 export YARN_SITE="/myriad-conf/yarn-site.xml"
@@ -25,7 +51,5 @@ if [ -z $HADOOP_NAMENODE ]; then
     exit 1
 fi
 
-# Ensure that the hadoop user is specified
-HADOOP_USER=${HADOOP_USER:='hduser'}
-
-su - $HADOOP_USER /usr/local/hadoop/bin/yarn resourcemanager
+#su - $HADOOP_USER /usr/local/hadoop/bin/yarn resourcemanager
+/usr/local/hadoop/bin/yarn resourcemanager


Mime
View raw message