bigtop-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From evan...@apache.org
Subject bigtop git commit: BIGTOP-1908. Move bigtop-deploy's dockerfiles into a centralized docker place
Date Fri, 23 Oct 2015 18:40:39 GMT
Repository: bigtop
Updated Branches:
  refs/heads/master 8bea68ff0 -> 755896bbd


BIGTOP-1908. Move bigtop-deploy's dockerfiles into a centralized docker place


Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo
Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/755896bb
Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/755896bb
Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/755896bb

Branch: refs/heads/master
Commit: 755896bbd9913608deb414564ecb961ddf2619d0
Parents: 8bea68f
Author: Evans Ye <evansye@apache.org>
Authored: Tue Sep 22 18:12:59 2015 +0000
Committer: Evans Ye <evansye@apache.org>
Committed: Sat Oct 24 02:40:10 2015 +0800

----------------------------------------------------------------------
 .../vm/vagrant-puppet-docker/README.md          | 39 +++++---------------
 .../vm/vagrant-puppet-docker/Vagrantfile        | 13 -------
 .../vm/vagrant-puppet-docker/centos/Dockerfile  | 33 -----------------
 .../vm/vagrant-puppet-docker/debian/Dockerfile  | 33 -----------------
 .../vm/vagrant-puppet-docker/docker-hadoop.sh   | 29 +++------------
 docker/bigtop-deploy/centos-6/Dockerfile        | 34 +++++++++++++++++
 docker/bigtop-deploy/centos-6/build.sh          | 18 +++++++++
 docker/bigtop-deploy/debian-8/Dockerfile        | 34 +++++++++++++++++
 docker/bigtop-deploy/debian-8/build.sh          | 18 +++++++++
 9 files changed, 119 insertions(+), 132 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/bigtop-deploy/vm/vagrant-puppet-docker/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/README.md b/bigtop-deploy/vm/vagrant-puppet-docker/README.md
index 6d609b6..79e50f7 100644
--- a/bigtop-deploy/vm/vagrant-puppet-docker/README.md
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/README.md
@@ -57,74 +57,55 @@ ssh into a docker container!
 
 ```
 service docker restart
-docker pull bigtop/seed:centos-6.4
+docker pull bigtop/deploy:centos-6
 ```
-
 Now, you can start your cluster:
 
-```
-cd bigtop/bigtop-deploy/vm/vagrant-puppet-docker
-./docker-hadoop.sh --build-image --create 3
-```
 In case of errors you can attempt running as root, or else, ping the mailing list.
 
 ## USAGE
 
-1) Build up the base Docker image that supports Vagrant.
-
-```
-./docker-hadoop.sh --build-image
-```
-
-2) Create a Bigtop Hadoop cluster by given # of node. (will place a file called config.rb)
+1) Create a Bigtop Hadoop cluster by given # of node. (will place a file called config.rb)
 
 ```
 ./docker-hadoop.sh --create 3
 ```
 
-3) Destroy the cluster.
+2) Destroy the cluster.
 
 ```
 ./docker-hadoop.sh --destroy
 ```
 
-4) Update your cluster after doing configuration changes. (re-run puppet apply)
+3) Update your cluster after doing configuration changes. (re-run puppet apply)
 
 ```
 ./docker-hadoop.sh --provision
 ```
 
-5) Run Bigtop smoke tests
+4) Run Bigtop smoke tests
 
 ```
 ./docker-hadoop.sh --smoke-tests
 ```
 
-6) Chain your operations with-in one command.
+5) Chain your operations with-in one command.
 
 ```
-./docker-hadoop.sh --build-image --create 5 --smoke-tests --destroy
+./docker-hadoop.sh --create 5 --smoke-tests --destroy
 ```
 
 Commands will be executed by following order:
 
 ```
-build-image => create 5 node cluster => destroy the cluster
-```
-
-7) Run hbase-test.sh to evaluate the deployment.
-
-```
-../utils/hbase-test.sh
+create 5 node cluster => run smoke tests => destroy the cluster
 ```
 
-8) See helper message:
+6) See helper message:
 
 ```
 ./docker-hadoop.sh -h
 usage: docker-hadoop.sh [options]
-       -b, --build-image                         Build base Docker image for Bigtop Hadoop
-                                                 (must be exectued at least once before creating
cluster)
        -c NUM_INSTANCES, --create=NUM_INSTANCES  Create a docker based Bigtop Hadoop cluster
        -p, --provision                           Deploy configuration changes
        -s, --smoke-tests                         Run Bigtop smoke tests
@@ -149,7 +130,7 @@ docker:
 
 ```
 boot2docker:
-        memory_size: "4196"
+        memory_size: "4096"
         number_cpus: "2"
 ```
 

http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile b/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile
index cc910b8..4927501 100755
--- a/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile
@@ -40,19 +40,6 @@ Vagrant.require_version ">= 1.6.0"
 VAGRANTFILE_API_VERSION = "2"
 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
 
-  # image definition
-  config.vm.define "image", autostart: false do |image|
-    image.vm.provider "docker" do |d|
-      d.build_dir = CONF["distro"]
-      d.build_args = ["-t",CONF["docker"]["image"]]
-      d.vagrant_machine = "boot2docker"
-      d.vagrant_vagrantfile = "./boot2docker/Vagrantfile"
-      d.has_ssh = false # We have decided to pipe commands to docker instead of ssh, due
to ssh hang issue in vagrant.
-    end
-    image.ssh.username = "root"
-    image.ssh.port = 22
-  end
-
   # nodes definition
   (1..$num_instances).each do |i|
     config.vm.define "bigtop#{i}" do |bigtop|

http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/bigtop-deploy/vm/vagrant-puppet-docker/centos/Dockerfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/centos/Dockerfile b/bigtop-deploy/vm/vagrant-puppet-docker/centos/Dockerfile
deleted file mode 100644
index d5d1f09..0000000
--- a/bigtop-deploy/vm/vagrant-puppet-docker/centos/Dockerfile
+++ /dev/null
@@ -1,33 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-FROM bigtop/seed:centos-6.4
-
-# enable ssh
-RUN yum -y install openssh-server openssh-clients sudo
-RUN sed -i.bak s/UsePAM\ yes/UsePAM\ no/ /etc/ssh/sshd_config
-RUN ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key
-RUN ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key
-
-# requiretty off
-RUN sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers
-
-# setup vagrant account
-RUN mkdir /root/.ssh/
-RUN chmod 0750 /root/.ssh
-RUN wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate
-O /root/.ssh/authorized_keys
-RUN chmod 0640 /root/.ssh/authorized_keys
-
-CMD /usr/sbin/sshd -D

http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/bigtop-deploy/vm/vagrant-puppet-docker/debian/Dockerfile
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/debian/Dockerfile b/bigtop-deploy/vm/vagrant-puppet-docker/debian/Dockerfile
deleted file mode 100644
index 2e089c3..0000000
--- a/bigtop-deploy/vm/vagrant-puppet-docker/debian/Dockerfile
+++ /dev/null
@@ -1,33 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-FROM debian:jessie
-
-# enable ssh
-
-RUN apt-get update
-RUN apt-get install -y openssh-server sudo wget
-
-# requiretty off
-RUN sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers
-
-# setup vagrant account
-RUN mkdir -p /root/.ssh
-RUN chmod 0750 /root/.ssh
-RUN wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate
-O /root/.ssh/authorized_keys
-RUN chmod 0640 /root/.ssh/authorized_keys
-
-RUN mkdir -p /var/run/sshd
-CMD /usr/sbin/sshd -D

http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh
----------------------------------------------------------------------
diff --git a/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh b/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh
index 626a408..9e30521 100755
--- a/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh
+++ b/bigtop-deploy/vm/vagrant-puppet-docker/docker-hadoop.sh
@@ -1,4 +1,5 @@
 #!/bin/bash
+
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
 # this work for additional information regarding copyright ownership.
@@ -18,8 +19,6 @@ usage() {
     echo "usage: $PROG [-C file ] args"
     echo "       -C file                                   Use alternate file for vagrantconfig.yaml"
     echo "  commands:"
-    echo "       -b, --build-image                         Build base Docker image for Bigtop
Hadoop"
-    echo "                                                 (must be exectued at least once
before creating cluster)"
     echo "       -c NUM_INSTANCES, --create=NUM_INSTANCES  Create a Docker based Bigtop Hadoop
cluster"
     echo "       -p, --provision                           Deploy configuration changes"
     echo "       -s, --smoke-tests                         Run Bigtop smoke tests"
@@ -28,17 +27,6 @@ usage() {
     exit 1
 }
 
-build-image() {
-    echo "\$vagrantyamlconf = \"$vagrantyamlconf\"" > config.rb
-    vagrant up image --provider docker
-    {
-        echo "echo -e '\nBUILD IMAGE SUCCESS.\n'" |vagrant ssh image
-    } || {
-        >&2 echo -e "\nBUILD IMAGE FAILED!\n"
-	exit 2
-    }
-}
-
 create() {
     echo "\$num_instances = $1" > config.rb
     echo "\$vagrantyamlconf = \"$vagrantyamlconf\"" >> config.rb
@@ -47,7 +35,7 @@ create() {
         echo "Docker container(s) startup failed!";
 	exit 1;
     fi
-    nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
+    nodes=(`vagrant status |grep bigtop |awk '{print $1}'`)
     hadoop_head_node=(`echo "hostname -f" |vagrant ssh ${nodes[0]} |tail -n 1`)
     repo=$(get-yaml-config repo)
     components="[`echo $(get-yaml-config components) | sed 's/ /, /g'`]"
@@ -73,7 +61,7 @@ create() {
 }
 
 provision() {
-    nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
+    nodes=(`vagrant status |grep bigtop |awk '{print $1}'`)
     for node in ${nodes[*]}; do
         bigtop-puppet $node &
     done
@@ -81,19 +69,15 @@ provision() {
 }
 
 smoke-tests() {
-    nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
+    nodes=(`vagrant status |grep bigtop |awk '{print $1}'`)
     smoke_test_components="`echo $(get-yaml-config smoke_test_components) | sed 's/ /,/g'`"
     echo "/bigtop-home/bigtop-deploy/vm/utils/smoke-tests.sh \"$smoke_test_components\""
|vagrant ssh ${nodes[0]}
 }
 
 
 destroy() {
-    nodes=(`vagrant status |grep running |grep -v image |awk '{print $1}'`)
+    vagrant destroy -f
     rm -rvf ./hosts ./config.rb
-    for node in ${nodes[*]}; do
-        vagrant destroy -f $node
-    done
-    wait
 }
 
 bigtop-puppet() {
@@ -120,9 +104,6 @@ fi
 vagrantyamlconf="vagrantconfig.yaml"
 while [ $# -gt 0 ]; do
     case "$1" in
-    -b|--build-image)
-        build-image
-        shift;;
     -c|--create)
         if [ $# -lt 2 ]; then
           echo "Create requires a number" 1>&2

http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/docker/bigtop-deploy/centos-6/Dockerfile
----------------------------------------------------------------------
diff --git a/docker/bigtop-deploy/centos-6/Dockerfile b/docker/bigtop-deploy/centos-6/Dockerfile
new file mode 100644
index 0000000..823dfc3
--- /dev/null
+++ b/docker/bigtop-deploy/centos-6/Dockerfile
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM bigtop/puppet:centos-6
+
+COPY bigtop_toolchain /etc/puppet/modules/bigtop_toolchain
+
+# enable ssh
+RUN yum -y install openssh-server openssh-clients vim && \
+    ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key && \
+    ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key && \
+    # requiretty off
+    sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers && \
+    # setup vagrant account
+    mkdir /root/.ssh && \
+    chmod 0700 /root/.ssh && \
+    wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate
-O /root/.ssh/authorized_keys && \
+    chmod 0600 /root/.ssh/authorized_keys && \
+    # Install required Puppet modules
+    puppet apply -e "include bigtop_toolchain::puppet-modules"
+
+CMD /usr/sbin/sshd -D

http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/docker/bigtop-deploy/centos-6/build.sh
----------------------------------------------------------------------
diff --git a/docker/bigtop-deploy/centos-6/build.sh b/docker/bigtop-deploy/centos-6/build.sh
new file mode 100755
index 0000000..c9a22c8
--- /dev/null
+++ b/docker/bigtop-deploy/centos-6/build.sh
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+cp -r ../../../bigtop_toolchain .
+docker build -t bigtop/deploy:centos-6  .
+rm -rf bigtop_toolchain

http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/docker/bigtop-deploy/debian-8/Dockerfile
----------------------------------------------------------------------
diff --git a/docker/bigtop-deploy/debian-8/Dockerfile b/docker/bigtop-deploy/debian-8/Dockerfile
new file mode 100644
index 0000000..bd0e8d3
--- /dev/null
+++ b/docker/bigtop-deploy/debian-8/Dockerfile
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM bigtop/puppet:debian-8
+
+COPY bigtop_toolchain /etc/puppet/modules/bigtop_toolchain
+
+# enable ssh
+RUN apt-get update && \
+    apt-get install -y openssh-server vim && \
+    mkdir -p /var/run/sshd && \
+    # requiretty off
+    sed -i.bak 's/requiretty/!requiretty/' /etc/sudoers && \
+    # setup vagrant account
+    mkdir -p /root/.ssh && \
+    chmod 0700 /root/.ssh && \
+    wget http://github.com/mitchellh/vagrant/raw/master/keys/vagrant.pub --no-check-certificate
-O /root/.ssh/authorized_keys && \
+    chmod 0600 /root/.ssh/authorized_keys && \
+    # Install required Puppet modules
+    puppet apply -e "include bigtop_toolchain::puppet-modules"
+
+CMD /usr/sbin/sshd -D

http://git-wip-us.apache.org/repos/asf/bigtop/blob/755896bb/docker/bigtop-deploy/debian-8/build.sh
----------------------------------------------------------------------
diff --git a/docker/bigtop-deploy/debian-8/build.sh b/docker/bigtop-deploy/debian-8/build.sh
new file mode 100755
index 0000000..e16d3ba
--- /dev/null
+++ b/docker/bigtop-deploy/debian-8/build.sh
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+cp -r ../../../bigtop_toolchain .
+docker build -t bigtop/deploy:debian-8 .
+rm -rf bigtop_toolchain


Mime
View raw message