hbase-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dimaspi...@apache.org
Subject hbase git commit: HBASE-18418 Remove apache_hbase_topology from dev-support
Date Wed, 18 Oct 2017 21:12:13 GMT
Repository: hbase
Updated Branches:
  refs/heads/master c16eb7881 -> 3acb08178


HBASE-18418 Remove apache_hbase_topology from dev-support


Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/3acb0817
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/3acb0817
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/3acb0817

Branch: refs/heads/master
Commit: 3acb081787a4289d86d977db26bedaf6a42172ce
Parents: c16eb78
Author: Dima Spivak <dimaspivak@apache.org>
Authored: Thu Jul 20 10:08:11 2017 -0700
Committer: Dima Spivak <dimaspivak@apache.org>
Committed: Wed Oct 18 14:08:26 2017 -0700

----------------------------------------------------------------------
 dev-support/apache_hbase_topology/Dockerfile    |  24 --
 dev-support/apache_hbase_topology/README.md     |  49 ---
 dev-support/apache_hbase_topology/__init__.py   |  15 -
 dev-support/apache_hbase_topology/actions.py    | 421 -------------------
 .../apache_hbase_topology/configurations.cfg    |  80 ----
 dev-support/apache_hbase_topology/profile.cfg   |  82 ----
 dev-support/apache_hbase_topology/ssh/id_rsa    |  44 --
 .../apache_hbase_topology/ssh/id_rsa.pub        |  18 -
 8 files changed, 733 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/Dockerfile
----------------------------------------------------------------------
diff --git a/dev-support/apache_hbase_topology/Dockerfile b/dev-support/apache_hbase_topology/Dockerfile
deleted file mode 100644
index 714a55c..0000000
--- a/dev-support/apache_hbase_topology/Dockerfile
+++ /dev/null
@@ -1,24 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-FROM debian:wheezy
-
-ENV TOPOLOGY_NAME=apache_hbase
-ADD . /root/clusterdock/clusterdock/topologies/${TOPOLOGY_NAME}
-
-RUN find /root -type f -name id_rsa -exec chmod 600 {} \;
-
-VOLUME /root/clusterdock/clusterdock/topologies/${TOPOLOGY_NAME}
-CMD ["/true"]

http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/README.md
----------------------------------------------------------------------
diff --git a/dev-support/apache_hbase_topology/README.md b/dev-support/apache_hbase_topology/README.md
deleted file mode 100644
index 018ee99..0000000
--- a/dev-support/apache_hbase_topology/README.md
+++ /dev/null
@@ -1,49 +0,0 @@
-<!---
-  Licensed to the Apache Software Foundation (ASF) under one
-  or more contributor license agreements.  See the NOTICE file
-  distributed with this work for additional information
-  regarding copyright ownership.  The ASF licenses this file
-  to you under the Apache License, Version 2.0 (the
-  "License"); you may not use this file except in compliance
-  with the License.  You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-  Unless required by applicable law or agreed to in writing,
-  software distributed under the License is distributed on an
-  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-  KIND, either express or implied.  See the License for the
-  specific language governing permissions and limitations
-  under the License.
--->
-# apache_hbase clusterdock topology
-
-## Overview
-*clusterdock* is a framework for creating Docker-based container clusters. Unlike regular
Docker
-containers, which tend to run single processes and then exit once the process terminates,
these
-container clusters are characterized by the execution of an init process in daemon mode.
As such,
-the containers act more like "fat containers" or "light VMs;" entities with accessible IP
addresses
-which emulate standalone hosts.
-
-*clusterdock* relies upon the notion of a topology to define how clusters should be built
into
-images and then what to do with those images to start Docker container clusters.
-
-## Usage
-The *clusterdock* framework is designed to be run out of its own container while affecting
-operations on the host. To avoid problems that might result from incorrectly
-formatting this framework invocation, a Bash helper script (`clusterdock.sh`) can be sourced
on a
-host that has Docker installed. Afterwards, running any of the binaries intended to carry
-out *clusterdock* actions can be done using the `clusterdock_run` command.
-```
-wget https://raw.githubusercontent.com/cloudera/clusterdock/master/clusterdock.sh
-# ALWAYS INSPECT SCRIPTS FROM THE INTERNET BEFORE SOURCING THEM.
-source clusterdock.sh
-```
-
-Since the *clusterdock* framework itself lives outside of Apache HBase, an environmental
variable
-is used to let the helper script know where to find an image of the *apache_hbase* topology.
To
-start a four-node Apache HBase cluster with default versions, you would simply run
-```
-CLUSTERDOCK_TOPOLOGY_IMAGE=apache_hbase_topology_location clusterdock_run \
-    ./bin/start_cluster apache_hbase --secondary-nodes='node-{2..4}'
-```

http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/__init__.py
----------------------------------------------------------------------
diff --git a/dev-support/apache_hbase_topology/__init__.py b/dev-support/apache_hbase_topology/__init__.py
deleted file mode 100644
index 635f0d9..0000000
--- a/dev-support/apache_hbase_topology/__init__.py
+++ /dev/null
@@ -1,15 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.

http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/actions.py
----------------------------------------------------------------------
diff --git a/dev-support/apache_hbase_topology/actions.py b/dev-support/apache_hbase_topology/actions.py
deleted file mode 100644
index 26566e0..0000000
--- a/dev-support/apache_hbase_topology/actions.py
+++ /dev/null
@@ -1,421 +0,0 @@
-#!/usr/bin/env python
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""The actions module for the apache_hbase topology. The behavior to be carried out by the
-build_cluster and start_cluster clusterdock scripts are to be defined through the build and
-start functions, respectively.
-"""
-
-import logging
-import tarfile
-from ConfigParser import ConfigParser
-from os import EX_OK, listdir, makedirs, remove # pylint: disable=ungrouped-imports
-                                                # Follow convention of grouping from module
imports
-                                                # after normal imports.
-from os.path import exists, join
-from shutil import move
-from socket import getfqdn
-from sys import stdout
-from uuid import uuid4
-
-# pylint: disable=import-error
-# clusterdock topologies get access to the clusterdock package at run time, but their reference
-# to clusterdock modules will confuse pylint, so we have to disable it.
-
-import requests
-from docker import Client
-
-from clusterdock import Constants
-from clusterdock.cluster import Cluster, Node, NodeGroup
-from clusterdock.docker_utils import (build_image, get_clusterdock_container_id,
-                                      get_host_port_binding, is_image_available_locally,
pull_image)
-from clusterdock.utils import strip_components_from_tar, XmlConfiguration
-
-# We disable a couple of Pylint conventions because it assumes that module level variables
must be
-# named as if they're constants (which isn't the case here).
-logger = logging.getLogger(__name__) # pylint: disable=invalid-name
-logger.setLevel(logging.INFO)
-
-client = Client() # pylint: disable=invalid-name
-
-DEFAULT_APACHE_NAMESPACE = Constants.DEFAULT.apache_namespace # pylint: disable=no-member
-
-def _copy_container_folder_to_host(container_id, source_folder, destination_folder,
-                                   host_folder=None):
-    if not exists(destination_folder):
-        makedirs(destination_folder)
-    stream, _ = client.get_archive(container_id, source_folder)
-    tar_filename = join(destination_folder, 'container_folder.tar')
-    with open(tar_filename, 'wb') as file_descriptor:
-        file_descriptor.write(stream.read())
-    tar = tarfile.open(name=tar_filename)
-    tar.extractall(path=destination_folder, members=strip_components_from_tar(tar))
-    tar.close()
-    remove(tar_filename)
-    logger.info("Extracted container folder %s to %s.", source_folder,
-                host_folder if host_folder else destination_folder)
-
-def _create_configs_from_file(filename, cluster_config_dir, wildcards):
-    configurations = ConfigParser(allow_no_value=True)
-    configurations.read(filename)
-
-    for config_file in configurations.sections():
-        logger.info("Updating %s...", config_file)
-        # For XML configuration files, run things through XmlConfiguration.
-        if config_file.endswith('.xml'):
-            XmlConfiguration(
-                {item[0]: item[1].format(**wildcards)
-                 for item in configurations.items(config_file)}
-            ).write_to_file(join(cluster_config_dir, config_file))
-        # For everything else, recognize whether a line in the configuration should simply
be
-        # appended to the bottom of a file or processed in some way. The presence of +++
will
-        # lead to the evaluation of the following string through the end of the line.
-        else:
-            lines = []
-            for item in configurations.items(config_file):
-                if item[0].startswith('+++'):
-                    command = item[0].lstrip('+ ').format(**wildcards)
-
-                    # Yes, we use eval here. This is potentially dangerous, but intention.
-                    lines.append(str(eval(command))) # pylint: disable=eval-used
-                elif item[0] == "body":
-                    lines.append(item[1].format(**wildcards))
-                else:
-                    lines.append(item[0].format(**wildcards))
-            with open(join(cluster_config_dir, config_file), 'w') as conf:
-                conf.write("".join(["{0}\n".format(line) for line in lines]))
-
-# Keep track of some common web UI ports that we'll expose to users later (e.g. to allow
a user
-# to reach the HDFS NameNode web UI over the internet).
-HBASE_REST_SERVER_PORT = 8080
-NAMENODE_WEB_UI_PORT = 50070
-RESOURCEMANAGER_WEB_UI_PORT = 8088
-
-# When starting or building cluster, CLUSTERDOCK_VOLUME will be the root directory for persistent
-# files (note that this location will itself be in a Docker container's filesystem).
-CLUSTERDOCK_VOLUME = '/tmp/clusterdock'
-
-def start(args):
-    """This function will be executed when ./bin/start_cluster apache_hbase is invoked."""
-
-    # pylint: disable=too-many-locals
-    # Pylint doesn't want more than 15 local variables in a function; this one has 17. This
is about
-    # as low as I want to go because, while I can cheat and stuff unrelated things in a dictionary,
-    # that won't improve readability.
-
-    uuid = str(uuid4())
-    container_cluster_config_dir = join(CLUSTERDOCK_VOLUME, uuid, 'config')
-    makedirs(container_cluster_config_dir)
-
-    for mount in client.inspect_container(get_clusterdock_container_id())['Mounts']:
-        if mount['Destination'] == CLUSTERDOCK_VOLUME:
-            host_cluster_config_dir = join(mount['Source'], uuid, 'config')
-            break
-    else:
-        raise Exception("Could not find source of {0} mount.".format(CLUSTERDOCK_VOLUME))
-
-    # CLUSTERDOCK_VOLUME/uuid/config in the clusterdock container corresponds to
-    # host_cluster_config_dir on the Docker host.
-    logger.debug("Creating directory for cluster configuration files in %s...",
-                 host_cluster_config_dir)
-
-    # Generate the image name to use from the command line arguments passed in.
-    image = '/'.join(
-        [item
-         for item in [args.registry_url, args.namespace or DEFAULT_APACHE_NAMESPACE,
-                      "clusterdock:{os}_java-{java}_hadoop-{hadoop}_hbase-{hbase}".format(
-                          os=args.operating_system, java=args.java_version,
-                          hadoop=args.hadoop_version, hbase=args.hbase_version
-                      )]
-         if item]
-    )
-    if args.always_pull or not is_image_available_locally(image):
-        pull_image(image)
-
-    # Before starting the cluster, we create a throwaway container from which we copy
-    # configuration files back to the host. We also use this container to run an HBase
-    # command that returns the port of the HBase master web UI. Since we aren't running init
here,
-    # we also have to manually pass in JAVA_HOME as an environmental variable.
-    get_hbase_web_ui_port_command = ('/hbase/bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool
'
-                                     'hbase.master.info.port')
-    container_id = client.create_container(image=image, command=get_hbase_web_ui_port_command,
-                                           environment={'JAVA_HOME': '/java'})['Id']
-    logger.debug("Created temporary container (id: %s) from which to copy configuration files.",
-                 container_id)
-
-    # Actually do the copying of Hadoop configs...
-    _copy_container_folder_to_host(container_id, '/hadoop/etc/hadoop',
-                                   join(container_cluster_config_dir, 'hadoop'),
-                                   join(host_cluster_config_dir, 'hadoop'))
-
-    # ... and repeat for HBase configs.
-    _copy_container_folder_to_host(container_id, '/hbase/conf',
-                                   join(container_cluster_config_dir, 'hbase'),
-                                   join(host_cluster_config_dir, 'hbase'))
-
-    logger.info("The /hbase/lib folder on containers in the cluster will be volume mounted
"
-                "into %s...", join(host_cluster_config_dir, 'hbase-lib'))
-    _copy_container_folder_to_host(container_id, '/hbase/lib',
-                                   join(container_cluster_config_dir, 'hbase-lib'),
-                                   join(host_cluster_config_dir, 'hbase-lib'))
-
-    # Every node in the cluster will have a shared volume mount from the host for Hadoop
and HBase
-    # configuration files as well as the HBase lib folder.
-    shared_volumes = [{join(host_cluster_config_dir, 'hadoop'): '/hadoop/etc/hadoop'},
-                      {join(host_cluster_config_dir, 'hbase'): '/hbase/conf'},
-                      {join(host_cluster_config_dir, 'hbase-lib'): '/hbase/lib'}]
-
-    # Get the HBase master web UI port, stripping the newline the Docker REST API gives us.
-    client.start(container=container_id)
-    if client.wait(container=container_id) == EX_OK:
-        hbase_master_web_ui_port = client.logs(container=container_id).rstrip()
-        client.remove_container(container=container_id, force=True)
-    else:
-        raise Exception('Failed to remove HBase configuration container.')
-
-    # Create the Node objects. These hold the state of our container nodes and will be started
-    # at Cluster instantiation time.
-    primary_node = Node(hostname=args.primary_node[0], network=args.network,
-                        image=image, ports=[NAMENODE_WEB_UI_PORT,
-                                            hbase_master_web_ui_port,
-                                            RESOURCEMANAGER_WEB_UI_PORT,
-                                            HBASE_REST_SERVER_PORT],
-                        volumes=shared_volumes)
-    secondary_nodes = []
-    for hostname in args.secondary_nodes:
-        # A list of service directories will be used to name folders on the host and, appended
-        # with an index, in the container, as well (e.g. /data1/node-1/dfs:/dfs1).
-        service_directories = ['dfs', 'yarn']
-
-        # Every Node will have shared_volumes to let one set of configs on the host be propagated
-        # to every container. If --data-directories is specified, this will be appended to
allow
-        # containers to use multiple disks on the host.
-        volumes = shared_volumes[:]
-        if args.data_directories:
-            data_directories = args.data_directories.split(',')
-            volumes += [{join(data_directory, uuid, hostname, service_directory):
-                             "/{0}{1}".format(service_directory, i)}
-                        for i, data_directory in enumerate(data_directories, start=1)
-                        for service_directory in service_directories]
-        secondary_nodes.append(Node(hostname=hostname,
-                                    network=args.network,
-                                    image=image,
-                                    volumes=volumes))
-
-    Cluster(topology='apache_hbase',
-            node_groups=[NodeGroup(name='primary', nodes=[primary_node]),
-                         NodeGroup(name='secondary', nodes=secondary_nodes)],
-            network_name=args.network).start()
-
-    # When creating configs, pass in a dictionary of wildcards into create_configurations_from_file
-    # to transform placeholders in the configurations.cfg file into real values.
-    _create_configs_from_file(filename=args.configurations,
-                              cluster_config_dir=container_cluster_config_dir,
-                              wildcards={"primary_node": args.primary_node,
-                                         "secondary_nodes": args.secondary_nodes,
-                                         "all_nodes": args.primary_node + args.secondary_nodes,
-                                         "network": args.network})
-
-    # After creating configurations from the configurations.cfg file, update hdfs-site.xml
and
-    # yarn-site.xml to use the data directories passed on the command line.
-    if args.data_directories:
-        _update_config_for_data_dirs(
-            container_cluster_config_dir=container_cluster_config_dir,
-            data_directories=data_directories
-        )
-
-    if not args.dont_start_services:
-        _start_services(primary_node, hbase_master_web_ui_port=hbase_master_web_ui_port)
-
-def _update_config_for_data_dirs(container_cluster_config_dir, data_directories):
-    logger.info('Updating dfs.datanode.data.dir in hdfs-site.xml...')
-    hdfs_site_xml_filename = join(container_cluster_config_dir, 'hadoop', 'hdfs-site.xml')
-    hdfs_site_xml = XmlConfiguration(
-        properties={'dfs.datanode.data.dir':
-                    ','.join(["/dfs{0}".format(i)
-                              for i, _ in enumerate(data_directories, start=1)])},
-        source_file=hdfs_site_xml_filename
-    )
-    hdfs_site_xml.write_to_file(filename=hdfs_site_xml_filename)
-
-    logger.info('Updating yarn.nodemanager.local-dirs in yarn-site.xml...')
-    yarn_site_xml_filename = join(container_cluster_config_dir, 'hadoop', 'yarn-site.xml')
-    yarn_site_xml = XmlConfiguration(
-        properties={'yarn.nodemanager.local-dirs':
-                    ','.join(["/yarn{0}".format(i)
-                              for i, _ in enumerate(data_directories, start=1)])},
-        source_file=yarn_site_xml_filename
-    )
-    yarn_site_xml.write_to_file(filename=yarn_site_xml_filename)
-
-def _start_services(primary_node, **kwargs):
-    logger.info("Formatting namenode on %s...", primary_node.fqdn)
-    primary_node.ssh('hdfs namenode -format')
-
-    logger.info("Starting HDFS...")
-    primary_node.ssh('/hadoop/sbin/start-dfs.sh')
-
-    logger.info("Starting YARN...")
-    primary_node.ssh('/hadoop/sbin/start-yarn.sh')
-
-    logger.info('Starting HBase...')
-    primary_node.ssh('/hbase/bin/start-hbase.sh')
-    primary_node.ssh('/hbase/bin/hbase-daemon.sh start rest')
-
-    logger.info("NameNode and HBase master are located on %s. SSH over and have fun!",
-                primary_node.hostname)
-
-    logger.info("The HDFS NameNode web UI can be reached at http://%s:%s",
-                getfqdn(), get_host_port_binding(primary_node.container_id,
-                                                 NAMENODE_WEB_UI_PORT))
-
-    logger.info("The YARN ResourceManager web UI can be reached at http://%s:%s",
-                getfqdn(), get_host_port_binding(primary_node.container_id,
-                                                 RESOURCEMANAGER_WEB_UI_PORT))
-
-    logger.info("The HBase master web UI can be reached at http://%s:%s",
-                getfqdn(), get_host_port_binding(primary_node.container_id,
-                                                 kwargs.get('hbase_master_web_ui_port')))
-
-    logger.info("The HBase REST server can be reached at http://%s:%s",
-                getfqdn(), get_host_port_binding(primary_node.container_id,
-                                                 HBASE_REST_SERVER_PORT))
-
-def build(args):
-    """This function will be executed when ./bin/build_cluster apache_hbase is invoked."""
-
-    # pylint: disable=too-many-locals
-    # See start function above for rationale for disabling this warning.
-
-    container_build_dir = join(CLUSTERDOCK_VOLUME, str(uuid4()))
-    makedirs(container_build_dir)
-
-    # If --hbase-git-commit is specified, we build HBase from source.
-    if args.hbase_git_commit:
-        build_hbase_commands = [
-            "git clone https://github.com/apache/hbase.git {0}".format(container_build_dir),
-            "git -C {0} checkout {1}".format(container_build_dir, args.hbase_git_commit),
-            "mvn --batch-mode clean install -DskipTests assembly:single -f {0}/pom.xml".format(
-                container_build_dir
-            )
-        ]
-
-        maven_image = Constants.docker_images.maven # pylint: disable=no-member
-        if not is_image_available_locally(maven_image):
-            pull_image(maven_image)
-
-        container_configs = {
-            'command': 'bash -c "{0}"'.format(' && '.join(build_hbase_commands)),
-            'image': maven_image,
-            'host_config': client.create_host_config(volumes_from=get_clusterdock_container_id())
-        }
-
-        maven_container_id = client.create_container(**container_configs)['Id']
-        client.start(container=maven_container_id)
-        for line in client.logs(container=maven_container_id, stream=True):
-            stdout.write(line)
-            stdout.flush()
-
-        # Mimic docker run --rm by blocking on docker wait and then removing the container
-        # if it encountered no errors.
-        if client.wait(container=maven_container_id) == EX_OK:
-            client.remove_container(container=maven_container_id, force=True)
-        else:
-            raise Exception('Error encountered while building HBase.')
-
-        assembly_target_dir = join(container_build_dir, 'hbase-assembly', 'target')
-        for a_file in listdir(assembly_target_dir):
-            if a_file.endswith('bin.tar.gz'):
-                args.hbase_tarball = join(assembly_target_dir, a_file)
-                break
-
-    # Download all the binary tarballs into our temporary directory so that we can add them
-    # into the Docker image we're building.
-    filenames = []
-    for tarball_location in [args.java_tarball, args.hadoop_tarball, args.hbase_tarball]:
-        tarball_filename = tarball_location.rsplit('/', 1)[-1]
-        filenames.append(tarball_filename)
-
-        # Download tarballs given as URLs.
-        if container_build_dir not in tarball_location:
-            get_request = requests.get(tarball_location, stream=True, cookies=(
-                {'oraclelicense': 'accept-securebackup-cookie'}
-                if tarball_location == args.java_tarball
-                else None
-            ))
-            # Raise Exception if download failed.
-            get_request.raise_for_status()
-            logger.info("Downloading %s...", tarball_filename)
-            with open(join(container_build_dir, tarball_filename), 'wb') as file_descriptor:
-                for chunk in get_request.iter_content(1024):
-                    file_descriptor.write(chunk)
-        else:
-            move(tarball_location, container_build_dir)
-
-    dockerfile_contents = r"""
-    FROM {nodebase_image}
-    COPY {java_tarball} /tarballs/
-    RUN mkdir /java && tar -xf /tarballs/{java_tarball} -C /java --strip-components=1
-    RUN echo "JAVA_HOME=/java" >> /etc/environment
-
-    COPY {hadoop_tarball} /tarballs/
-    RUN mkdir /hadoop && tar -xf /tarballs/{hadoop_tarball} -C /hadoop --strip-components=1
-    COPY {hbase_tarball} /tarballs/
-    RUN mkdir /hbase && tar -xf /tarballs/{hbase_tarball} -C /hbase --strip-components=1
-
-    # Remove tarballs folder.
-    RUN rm -rf /tarballs
-
-    # Set PATH explicitly.
-    RUN echo "PATH=/java/bin:/hadoop/bin:/hbase/bin/:$(echo $PATH)" >> /etc/environment
-
-    # Add hbase user and group before copying root's SSH keys over.
-    RUN groupadd hbase \
-        && useradd -g hbase hbase \
-        && cp -R /root/.ssh ~hbase \
-        && chown -R hbase:hbase ~hbase/.ssh
-
-    # Disable requiretty in /etc/sudoers as required by HBase chaos monkey.
-    RUN sed -i 's/Defaults\s*requiretty/#&/' /etc/sudoers
-    """.format(nodebase_image='/'.join([item
-                                        for item in [args.registry_url,
-                                                     args.namespace or DEFAULT_APACHE_NAMESPACE,
-                                                     "clusterdock:{os}_nodebase".format(
-                                                         os=args.operating_system
-                                                     )]
-                                        if item]),
-               java_tarball=filenames[0], hadoop_tarball=filenames[1], hbase_tarball=filenames[2])
-
-    logger.info("Created Dockerfile: %s", dockerfile_contents)
-
-    with open(join(container_build_dir, 'Dockerfile'), 'w') as dockerfile:
-        dockerfile.write(dockerfile_contents)
-
-    image = '/'.join(
-        [item
-         for item in [args.registry_url, args.namespace or DEFAULT_APACHE_NAMESPACE,
-                      "clusterdock:{os}_java-{java}_hadoop-{hadoop}_hbase-{hbase}".format(
-                          os=args.operating_system, java=args.java_version,
-                          hadoop=args.hadoop_version, hbase=args.hbase_version
-                      )]
-         if item])
-
-    logger.info("Building image %s...", image)
-    build_image(dockerfile=join(container_build_dir, 'Dockerfile'), tag=image)
-
-    logger.info("Removing build temporary directory...")
-    return [image]

http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/configurations.cfg
----------------------------------------------------------------------
diff --git a/dev-support/apache_hbase_topology/configurations.cfg b/dev-support/apache_hbase_topology/configurations.cfg
deleted file mode 100644
index f28995e..0000000
--- a/dev-support/apache_hbase_topology/configurations.cfg
+++ /dev/null
@@ -1,80 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# This configuration file is used by the apache-hbase clusterdock topology to populate configuration
-# files for an HBase cluster. Section names denote the filenames (e.g. [hadoop/core-site.xml])
with
-# the section's corresponding items denoting properties.
-#
-# Filenames ending with .xml will have items run through an XML parser. That is,
-#
-# [hbase/hbase-site.xml]
-# hbase.cluster.distributed = true
-#
-# would lead to the creation of an hbase-site.xml file containing:
-#
-# <property>
-#   <name>hbase.cluster.distributed</name>
-#   <value>true</value>
-# </property>
-#
-# Note, also, that items in non-xml files can be copied verbatim by with using the "body:"
item (be
-# sure to include leading whitespace must be used for the following lines). For example:
-#
-# [hbase/hbase-env.sh]
-# body:
-#     COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+UseG1GC"
-#     COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintGCDetails"
-#
-# would result in an hbase-env.sh file with:
-#
-#     COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+UseG1GC"
-#     COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintGCDetails"
-#
-# Two last notes:
-# 1. Items starting with +++ will be eval'd with Python directly.
-# 2. As defined in action.py, some wildcards are processed at cluster start time (e.g. {network}).
-
-[hadoop/slaves]
-+++ '\n'.join(["{{0}}.{network}".format(node) for node in {secondary_nodes}])
-
-[hadoop/core-site.xml]
-fs.default.name = hdfs://{primary_node[0]}.{network}:8020
-
-[hadoop/mapred-site.xml]
-mapreduce.framework.name = yarn
-
-[hadoop/yarn-site.xml]
-yarn.resourcemanager.hostname = {primary_node[0]}.{network}
-yarn.nodemanager.aux-services = mapreduce_shuffle
-yarn.nodemanager.aux-services.mapreduce_shuffle.class = org.apache.hadoop.mapred.ShuffleHandler
-yarn.nodemanager.vmem-check-enabled = false
-
-[hbase/regionservers]
-+++ '\n'.join(["{{0}}.{network}".format(node) for node in {secondary_nodes}])
-
-[hbase/backup-masters]
-{secondary_nodes[0]}.{network}
-
-[hbase/hbase-site.xml]
-hbase.cluster.distributed = true
-hbase.rootdir = hdfs://{primary_node[0]}.{network}/hbase
-hbase.zookeeper.quorum = {primary_node[0]}.{network}
-hbase.zookeeper.property.dataDir = /usr/local/zookeeper
-
-# For now, set service users for Chaos Monkey to be root.
-hbase.it.clustermanager.hadoop.hdfs.user = root
-hbase.it.clustermanager.zookeeper.user = root
-hbase.it.clustermanager.hbase.user = root

http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/profile.cfg
----------------------------------------------------------------------
diff --git a/dev-support/apache_hbase_topology/profile.cfg b/dev-support/apache_hbase_topology/profile.cfg
deleted file mode 100644
index e6937ff..0000000
--- a/dev-support/apache_hbase_topology/profile.cfg
+++ /dev/null
@@ -1,82 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# This configuration file is used to define some general properties of the apache-hbase clusterdock
-# topology, as well as its command line.
-
-[general]
-name = Apache HBase
-description = An Apache HBase cluster with 1 primary node and n-1 secondary nodes
-
-
-[node_groups]
-# Define node groups and specify which of each to start during the build process.
-primary-node = node-1
-secondary-nodes = node-2
-
-
-[build]
-arg.java-tarball = http://download.oracle.com/otn-pub/java/jdk/8u91-b14/jdk-8u91-linux-x64.tar.gz
-arg.java-tarball.help = The URL (or filename) of the Java tarball to install on the cluster
-arg.java-tarball.metavar = tarball
-
-arg.java-version = 8u91
-arg.java-version.help = The label to use when identifying the version of Java
-arg.java-version.metavar = ver
-
-arg.hadoop-tarball = https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz
-arg.hadoop-tarball.help = The URL (or filename) of the Hadoop tarball to install on the cluster
-arg.hadoop-tarball.metavar = tarball
-
-arg.hadoop-version = 2.7.2
-arg.hadoop-version.help = The label to use when identifying the version of Hadoop
-arg.hadoop-version.metavar = ver
-
-arg.hbase-tarball
-arg.hbase-tarball.help = The URL (or filename) of the HBase tarball to install on the cluster
-arg.hbase-tarball.metavar = tarball
-
-arg.hbase-version
-arg.hbase-version.help = The label to use when identifying the version of HBase
-arg.hbase-version.metavar = ver
-
-arg.hbase-git-commit
-arg.hbase-git-commit.help = The git commit to checkout when building the image
-arg.hbase-git-commit.metavar = commit
-
-[start]
-arg.java-version = 8u91
-arg.java-version.help = The Java version on the cluster
-arg.java-version.metavar = ver
-
-arg.hadoop-version = 2.7.2
-arg.hadoop-version.help = The Hadoop version on the cluster
-arg.hadoop-version.metavar = ver
-
-arg.hbase-version = master
-arg.hbase-version.help = The HBase version on the cluster
-arg.hbase-version.metavar = ver
-
-arg.configurations = /root/clusterdock/clusterdock/topologies/apache_hbase/configurations.cfg
-arg.configurations.help = Location of a configurations.cfg file to apply to the cluster
-arg.configurations.metavar = path
-
-arg.data-directories
-arg.data-directories.help = A comma-separated list of host directories on which to save data
(i.e. for HDFS and YARN)
-arg.data-directories.metavar = dir
-
-arg.dont-start-services = False
-arg.dont-start-services.help = Don't start Hadoop and HBase services as part of cluster start

http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/ssh/id_rsa
----------------------------------------------------------------------
diff --git a/dev-support/apache_hbase_topology/ssh/id_rsa b/dev-support/apache_hbase_topology/ssh/id_rsa
deleted file mode 100644
index 7c9d31e..0000000
--- a/dev-support/apache_hbase_topology/ssh/id_rsa
+++ /dev/null
@@ -1,44 +0,0 @@
-#/**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements.  See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership.  The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License.  You may obtain a copy of the License at
-# *
-# *     http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
------BEGIN RSA PRIVATE KEY-----
-MIIEpAIBAAKCAQEAtj/yZZNF+bv26eqWqsx+vFehSlxBJp/QhIrFWKpjHcpQJ29o
-6hJN9moU3Goft2C5w6FoZdjC1TWlzhxUnRS8xeksFnW3ItvkjySLA1Iq6jchYxNd
-fZ3HwTdH0rubM1uJ/CnkaijoxBqGBmPSL0TxfqcteaJ8APhslwl0WWJ6b+tBCHDV
-pTLATebtggCAfhKmSuAYmn3QIqJ7DFoSGkwhkxpUHuuVCZxUH3CIxLIw+6npArr/
-S5gtFo50oi6FXPmvv6mJg6yLqq3VlKcQh6d/COaJopHn+nLed2kECESUlpTruMpr
-6IcGESgz4hnkmhop8oTY42sQJhPPF2Ahq9a3aQIDAQABAoIBADSbxPb5SkvKrH3d
-j9yB51uq2A5FDzF9FI4OGOV9WdsxmW2oxVo8KnElMhxmLf2bWERWhXJQ3fz53YDf
-wLUPVWaz5lwdYt4XJ6UCYXZ185lkjKiy4FvwfccSlBMKwMRUekJmPV8/q+Ff3qxd
-iEDI4AU1cPUZqD4HeCEpQ4LB4KIJhCdLkCgoWxxaCwwuB6DnwB4P4VLeAfsm2NEX
-k9dld87q/miOmuw9QsmSv9wYiQqoPdV5Qj7KYqHBAa6syqUfFni3Ibmw1WBzMydp
-8YyP9HvrzDBMnPPzkmp6od0fAgGafIlkIxz/9sCKOSISnuuqahbNAJK/rIiJzLY3
-Pi49M+ECgYEA2vCFObmM/hpKUPNxG841nQScFfeMg1q1z1krEmfjqDTELXyq9AOS
-fGiVTxfWagOGoAWIwg3ZfgGEmxsKrOSxkFvabWdhN1Wn98Zf8fJG8YAwLYg8JOgf
-gZ5pkxFW4FwrAeFDyJyKvFJVrbDw1PM41yvTmRzf3NjcaqJrBE2fgKUCgYEA1RmF
-XjfMlBoMLZ4pXS1zF91WgOF4SNdJJj9RCGzqqdy+DzPKNAZVa0HBhaIHCZXL3Hcv
-zqgEb6RSMysyVYjZPwfSwevsSuxpfriVpYux5MN3AEXX5Ysv51WWStWgt9+iQlfo
-xAdxxukOa++PZ4Z+TIIEDAFS47rnKEQUh+ZNfHUCgYEA0amTa3wtcQlsMalvn9kR
-rpRDhSXTAddUVIRnovCqKuKdG5JPg+4H0eu1UFDbnBpUSdoC5RKuPOTnQEHdL0Sy
-ZjQQMMTXbE4y1Cy8pM4G8i539KKKNi20PkSdhaENOT4KUXqPlwWSNlYChprzhnqE
-7EmkEPR9zNg//D4djbloDaECgYANOJIfsFKO9ba/tcpXL5SubFsLj/GIg2LUbqU2
-YpuEgl+ATfRDmgj+qIu7ILxTCeol+XcL2Ty9OHKpHgr3Z5Ai6vdWdK6qT1SUOht+
-s9YLnVzqtWqZoTMNpS+34N0hy0wj1ZRpZRTYBGmSpMA+6gc38/EQVZyw6E2jH+Yu
-MEmqaQKBgQDGh9uXCl/WjhBOF9VLrX/Aeaa2Mzrh21Ic1dw6aWrE4EW6k1LvSP36
-evrvrs2jQuzRMGH6DKX8ImnVEWjK+gZfgf2MuyDSW7KYR5zxkdZtRkotF6X0fu6N
-8uLa7CN8UmS4FiAMLwNbTJ6zA6ohny7r+AiOqNGlP9vBFMhpGs3NFg==
------END RSA PRIVATE KEY-----

http://git-wip-us.apache.org/repos/asf/hbase/blob/3acb0817/dev-support/apache_hbase_topology/ssh/id_rsa.pub
----------------------------------------------------------------------
diff --git a/dev-support/apache_hbase_topology/ssh/id_rsa.pub b/dev-support/apache_hbase_topology/ssh/id_rsa.pub
deleted file mode 100644
index 0dca44a..0000000
--- a/dev-support/apache_hbase_topology/ssh/id_rsa.pub
+++ /dev/null
@@ -1,18 +0,0 @@
-#/**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements.  See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership.  The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License.  You may obtain a copy of the License at
-# *
-# *     http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2P/Jlk0X5u/bp6paqzH68V6FKXEEmn9CEisVYqmMdylAnb2jqEk32ahTcah+3YLnDoWhl2MLVNaXOHFSdFLzF6SwWdbci2+SPJIsDUirqNyFjE119ncfBN0fSu5szW4n8KeRqKOjEGoYGY9IvRPF+py15onwA+GyXCXRZYnpv60EIcNWlMsBN5u2CAIB+EqZK4BiafdAionsMWhIaTCGTGlQe65UJnFQfcIjEsjD7qekCuv9LmC0WjnSiLoVc+a+/qYmDrIuqrdWUpxCHp38I5omikef6ct53aQQIRJSWlOu4ymvohwYRKDPiGeSaGinyhNjjaxAmE88XYCGr1rdp
clusterdock


Mime
View raw message