hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dima Spivak (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
Date Thu, 02 Jun 2016 02:29:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311607#comment-15311607
] 

Dima Spivak commented on HBASE-12721:
-------------------------------------

Please do! Here are some instructions to get started while I continue to hammer out a proper
patch:
# Install Docker 1.11 on a machine. I did the development and testing of this under an Ubuntu
14.04 AWS instance with a 3.13.0-85 kernel (earlier Ubuntu kernels might have problems), so
something similar would be the surest way not to run into issues. If you have a cloud instance
at your disposal, you can simplify setup by taking advantage of a framework script that I've
temporarily uploaded to Gist (I ran it through bit.ly because of the long raw URL, but I won't
be offended if you read through the script before running it). If using the script, spin up
your single cloud instance and then run {{curl -sL http://bit.ly/20UntA8}}, which will install
Docker, update the host's root user's SSH keys to let you log into the containers we're going
to start up, and generally make your life easier. *Note that this script copies the SSH keys
from the Docker images we're creating into the {{~root/.ssh folder}} on your host, so don't
use this script on a machine with an existing SSH keypair in {{~root/.ssh}}.*
# If you ran my setup script, run {{source /usr/local/bin/clusterdock.sh}} on the host machine
to enable use of {{clusterdock_run}}, a small helper function that makes the framework super
easy to use. _Instead of dealing with virtualenvs for this Python project, we run it out of
a Docker container_. If you already had Docker installed and just want to source the script
from Gist, run
{noformat}
curl -sL http://bit.ly/1ROTmE3 | source /dev/stdin
{noformat}
# Stand up an HBase cluster. For the purpose of a quick demo, I uploaded images I built with
this framework to my personal Docker Hub acount. To get a running 4-node HBase cluster with
Oracle Java 8u91, Hadoop 2.7.2, and the most recent build of HBase {{master}} as of when I
built it an hour ago, just type:
{noformat}
clusterdock_run ./bin/start_cluster apache_hbase --primary-node=node-1 --secondary-nodes='node-{2..4}'
--hbase-version=master --start-services
{noformat}

That's it. Running that will pull down the necessary HBase image (also built with this framework),
setup some configs (i.e. you could have created a 5-node cluster by changing the Bash-expandable
argument to {{\-\-secondary-nodes}}), and then start up services. Standard out will give information
on what's happening and how to access various web UIs, but you should also be able to SSH
to nodes by name (assuming you have the key-pairs setup) by using the hostnames shown during
startup. Other things to try:
- Stop and remove all running containers:
{noformat}
clusterdock_run ./bin/housekeeping nuke
{noformat}
- Build your own HBase cluster images:
{noformat}
clusterdock_run ./bin/build_cluster apache_hbase --hbase-version=master --hbase-git-commit=master
{noformat}
- Learn more about options a particular script provides:
{noformat}
clusterdock_run ./bin/start_cluster -h
{noformat}

Let me know if you run into any issues!

> Create Docker container cluster infrastructure to enable better testing
> -----------------------------------------------------------------------
>
>                 Key: HBASE-12721
>                 URL: https://issues.apache.org/jira/browse/HBASE-12721
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Dima Spivak
>            Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support as "hbase_docker;"
all this did was stand up a standalone cluster from source and start a shell. Now seems like
a good time to extend this to be useful for applications that could actual benefit the community,
especially around testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up distributed
HBase clusters on a single host machine in a couple minutes and run our awesome hbase-it suite
against it.
> - Binary compatibility testing of an HBase client is easiest when standing up an HBase
cluster can be done once and then different client source/binary permutations run against
it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any upstream
automation on build.apache.org, in part because it's a pain to set up x-node clusters on Apache
infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own top-level
module ("hbase-docker" would conveniently fit the existing schema :-)), strives to create
a simple framework for deploying "distributed," multi-container Apache HBase clusters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message