singa-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wang...@apache.org
Subject incubator-singa git commit: SINGA-150 Mesos Docker container failed
Date Thu, 24 Mar 2016 05:11:08 GMT
Repository: incubator-singa
Updated Branches:
  refs/heads/master 259422c41 -> a6eea9c4a


SINGA-150 Mesos Docker container failed

The initial ticket was raised when executing SINGA using Mesos containers threw exception
`called InitGoogleLoggin() twice`

This issue contains fixes for the initial problem, plus refactoring of the
Docker build process. A new Dockerfile is added to build GPU-enabled
SINGA. We also update the documentation for building SINGA containers,
since new versions of Docker have removed features that the we depended on.
Particularly, since Docker 1.9, launching new containers will not update
`/etc/hosts` files on the existing containers, causing name resolution to fail.


Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/a6eea9c4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/a6eea9c4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/a6eea9c4

Branch: refs/heads/master
Commit: a6eea9c4a5fe9713fbd5c5505c3bb13ccf79cbed
Parents: 259422c
Author: Anh Dinh <ug93tad@gmail.com>
Authored: Fri Mar 18 16:22:22 2016 +0800
Committer: Anh Dinh <ug93tad@gmail.com>
Committed: Wed Mar 23 23:03:51 2016 +0800

----------------------------------------------------------------------
 tool/docker/mesos/Dockerfile     |  9 ++---
 tool/docker/singa/Dockerfile     |  9 ++---
 tool/docker/singa/Dockerfile_gpu | 62 +++++++++++++++++++++++++++++++++++
 tool/mesos/singa_scheduler.cc    |  1 -
 4 files changed, 72 insertions(+), 9 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/a6eea9c4/tool/docker/mesos/Dockerfile
----------------------------------------------------------------------
diff --git a/tool/docker/mesos/Dockerfile b/tool/docker/mesos/Dockerfile
index fc5cca0..d2da1f3 100644
--- a/tool/docker/mesos/Dockerfile
+++ b/tool/docker/mesos/Dockerfile
@@ -22,8 +22,9 @@
 FROM singa/base
 MAINTAINER Anh Dinh<ug93tad@gmail.com>
 
-COPY .bashrc /root/.bashrc
-COPY install.sh /opt/install.sh
+COPY incubator-singa/tool/docker/mesos/.bashrc /root/.bashrc
+COPY incubator-singa/tool/docker/mesos/install.sh /opt/install.sh
+COPY incubator-singa/tool/mesos/* /root/incubator-singa/tool/mesos/
 
 RUN cd /opt && source ./install.sh
 
@@ -34,8 +35,8 @@ RUN cd /opt && source /root/.bashrc && wget -c http://www.eu.apache.org/dist/had
 #to be removed after SINGA-11 is merged (pulling from Anh's branch instead of pulling from
the master)
 #RUN source ~/.bashrc && mkdir /root/mesos && cd /root/mesos && git
clone https://github.com/ug93tad/incubator-singa && cd incubator-singa &&
git checkout SINGA-11 && cp -r tool/mesos /root/incubator-singa/tool/ && cd
/root/incubator-singa/tool/mesos && make 
 
+# update SINGA, but keep the examples directory 
 RUN source ~/.bashrc && cd /root/incubator-singa/tool/mesos && make
 
-COPY *.xml /opt/hadoop-2.6.0/etc/hadoop/
-
+COPY incubator-singa/tool/docker/mesos/*.xml /opt/hadoop-2.6.0/etc/hadoop/
 RUN ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa && cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
&& echo 'StrictHostKeyChecking no' >> ~/.ssh/config

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/a6eea9c4/tool/docker/singa/Dockerfile
----------------------------------------------------------------------
diff --git a/tool/docker/singa/Dockerfile b/tool/docker/singa/Dockerfile
index 9f542dc..07ca2cc 100644
--- a/tool/docker/singa/Dockerfile
+++ b/tool/docker/singa/Dockerfile
@@ -27,9 +27,10 @@ RUN apt-get update && apt-get -y install g++-4.8 build-essential
git vim wget zi
 
 RUN cd /opt && wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie"
 http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz &&
cd /opt && tar -zxvf jdk-8u60-linux-x64.tar.gz && rm -rf jdk-8u60-linux-x64.tar.gz
 
-COPY ssh.conf /etc/supervisor/conf.d/
-COPY .bashrc /root/.bashrc
-copy .vimrc /root/.vimrc
+COPY incubator-singa/tool/docker/singa/ssh.conf /etc/supervisor/conf.d/
+COPY incubator-singa/tool/docker/singa/.bashrc /root/.bashrc
+COPY incubator-singa/tool/docker/singa/.vimrc /root/.vimrc
+ADD incubator-singa /root/incubator-singa
 
-RUN cd && source /root/.bashrc && git clone https://github.com/apache/incubator-singa.git
&& cd incubator-singa && ./thirdparty/install.sh all && cd thirdparty/
&& wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
&& tar -zxvf protobuf-2.5.0.tar.gz && cd protobuf-2.5.0 && ./configure
&& make && make install && cd ../../ && cp thirdparty/install.sh
. && rm -rf thirdparty/* && mv install.sh thirdparty/ && ./thirdparty/install.sh
zookeeper && ./autogen.sh && ./configure && make && make install
&& cd examples/cifar10 && mv Makefile.example Makefile && make download
&& make create && service ssh start
+RUN cd && source /root/.bashrc && cd incubator-singa && ./thirdparty/install.sh
all && cd thirdparty/ && wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
&& tar -zxvf protobuf-2.5.0.tar.gz && cd protobuf-2.5.0 && ./configure
&& make clean && make && make install && cd ../../ &&
cp thirdparty/install.sh . && rm -rf thirdparty/* && mv install.sh thirdparty/
&& ./thirdparty/install.sh zookeeper && ./autogen.sh && ./configure
&& make && make install && cd examples/cifar10 && mv Makefile.example
Makefile && make download && make create && service ssh start
 

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/a6eea9c4/tool/docker/singa/Dockerfile_gpu
----------------------------------------------------------------------
diff --git a/tool/docker/singa/Dockerfile_gpu b/tool/docker/singa/Dockerfile_gpu
new file mode 100644
index 0000000..3c63b7d
--- /dev/null
+++ b/tool/docker/singa/Dockerfile_gpu
@@ -0,0 +1,62 @@
+FROM singa/base
+MAINTAINER NVIDIA CORPORATION <digits@nvidia.com>
+
+LABEL com.nvidia.volumes.needed="nvidia_driver"
+
+ENV NVIDIA_GPGKEY_SUM bd841d59a27a406e513db7d405550894188a4c1cd96bf8aa4f82f1b39e0b5c1c
+ENV NVIDIA_GPGKEY_FPR 889bee522da690103c4b085ed88c3d385c37d3be
+
+RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/GPGKEY
&& \
+    apt-key adv --export --no-emit-version -a $NVIDIA_GPGKEY_FPR | tail -n +2 > cudasign.pub
&& \
+    echo "$NVIDIA_GPGKEY_SUM  cudasign.pub" | sha256sum -c --strict - && rm cudasign.pub
&& \
+    echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64 /"
> /etc/apt/sources.list.d/cuda.list
+
+ENV CUDA_VERSION 7.0
+LABEL com.nvidia.cuda.version="7.0"
+
+ENV CUDA_PKG_VERSION 7-0=7.0-28
+RUN apt-get update && apt-get install -y --no-install-recommends --force-yes curl
\
+        cuda-nvrtc-$CUDA_PKG_VERSION \
+        cuda-cusolver-$CUDA_PKG_VERSION \
+        cuda-cublas-$CUDA_PKG_VERSION \
+        cuda-cufft-$CUDA_PKG_VERSION \
+        cuda-curand-$CUDA_PKG_VERSION \
+        cuda-cusparse-$CUDA_PKG_VERSION \
+        cuda-npp-$CUDA_PKG_VERSION \
+        cuda-cudart-$CUDA_PKG_VERSION && \
+    ln -s cuda-$CUDA_VERSION /usr/local/cuda 
+
+RUN apt-get install -y --no-install-recommends --force-yes \
+        cuda-core-$CUDA_PKG_VERSION \
+        cuda-misc-headers-$CUDA_PKG_VERSION \
+        cuda-command-line-tools-$CUDA_PKG_VERSION \
+        cuda-license-$CUDA_PKG_VERSION \
+        cuda-nvrtc-dev-$CUDA_PKG_VERSION \
+        cuda-cusolver-dev-$CUDA_PKG_VERSION \
+        cuda-cublas-dev-$CUDA_PKG_VERSION \
+        cuda-cufft-dev-$CUDA_PKG_VERSION \
+        cuda-curand-dev-$CUDA_PKG_VERSION \
+        cuda-cusparse-dev-$CUDA_PKG_VERSION \
+        cuda-npp-dev-$CUDA_PKG_VERSION \
+        cuda-cudart-dev-$CUDA_PKG_VERSION \
+        cuda-driver-dev-$CUDA_PKG_VERSION 
+
+RUN echo "/usr/local/cuda/lib" >> /etc/ld.so.conf.d/cuda.conf && \
+    echo "/usr/local/cuda/lib64" >> /etc/ld.so.conf.d/cuda.conf && \
+    ldconfig
+
+ENV CUDNN_DOWNLOAD_SUM 98679d5ec039acfd4d81b8bfdc6a6352d6439e921523ff9909d364e706275c2b
+
+RUN curl -fsSL http://developer.download.nvidia.com/compute/redist/cudnn/v3/cudnn-7.0-linux-x64-v3.0-prod.tgz
-O && \
+    echo "$CUDNN_DOWNLOAD_SUM  cudnn-7.0-linux-x64-v3.0-prod.tgz" | sha256sum -c --strict
- && \
+    tar -xzf cudnn-7.0-linux-x64-v3.0-prod.tgz -C /usr/local && \
+    rm cudnn-7.0-linux-x64-v3.0-prod.tgz && \
+    ldconfig
+
+RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \
+    echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
+
+ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
+ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64:${LD_LIBRARY_PATH}
+
+RUN cd && source /root/.bashrc && cd incubator-singa && ./configure
--enable-cuda --with-cuda=/usr/local/cuda --enable-cudnn --with-cudnn=/usr/local/cuda &&
make && make install 

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/a6eea9c4/tool/mesos/singa_scheduler.cc
----------------------------------------------------------------------
diff --git a/tool/mesos/singa_scheduler.cc b/tool/mesos/singa_scheduler.cc
index 408b609..c9b72d4 100644
--- a/tool/mesos/singa_scheduler.cc
+++ b/tool/mesos/singa_scheduler.cc
@@ -376,7 +376,6 @@ class SingaScheduler: public mesos::Scheduler {
 
 int main(int argc, char** argv) {
   FLAGS_logtostderr = 1;
-  google::InitGoogleLogging(argv[0]); 
   int status = mesos::DRIVER_RUNNING;
   SingaScheduler *scheduler;
   if (!(argc == 2 || argc == 4 || argc == 6)) {


Mime
View raw message