Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 3 Jun 2016 00:08:59 +0000 (UTC)
From: "Andrew Purtell (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12762646.1418926212000.21279.1464912539930@Atlassian.JIRA>
In-Reply-To: <JIRA.12762646.1418926212000@Atlassian.JIRA>
References: <JIRA.12762646.1418926212000@Atlassian.JIRA> <JIRA.12762646.1418926212552@arcas>
Subject: [jira] [Commented] (HBASE-12721) Create Docker container cluster
 infrastructure to enable better testing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 03 Jun 2016 00:09:01 -0000


    [ https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313360#comment-15313360 ] 

Andrew Purtell commented on HBASE-12721:
----------------------------------------

bq. clusterdock_run ./bin/build_cluster apache_hbase --hbase-version=0.98 --hbase-git-commit=0.98

Let me try this now. Really liking the ability to specify any git ref. 

bq. Once those images exist, you can pass the location of an .ini-like file as an argument to start_cluster and it will set configurations for you before starting services 

Ok, what I would like to do is control how hadoop-env.sh and hbase-env.sh are generated. Consider something like:

*hadoop-env.sh*
{noformat}
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+UseG1GC"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:MaxGCPauseMillis=100"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintGCDetails"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintGCDateStamps"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintGCTimeStamps"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintAdaptiveSizePolicy"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintReferenceGC"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+ParallelRefProcEnabled"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+TieredCompilation"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:-ResizePLAB"

export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS -Xms1g -Xmx1g"
export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS \$COMMON_HDFS_OPTS"
export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS -XX:+AlwaysPreTouch"
export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS -verbose:gc -Xloggc:/var/log/hadoop/hdfs-namenode-gc.log"

export HADOOP_SECONDARYNAMENODE_OPTS="\$HADOOP_SECONDARYNAMENODE_OPTS -Xms1g -Xmx1g"
export HADOOP_SECONDARYNAMENODE_OPTS="\$HADOOP_SECONDARYNAMENODE_OPTS \$COMMON_HDFS_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="\$HADOOP_SECONDARYNAMENODE_OPTS -verbose:gc -Xloggc:/var/log/hadoop/hdfs-secondarynamenode-gc.log"

export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS -Xms1g -Xmx1g"
export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS \$COMMON_HDFS_OPTS"
export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS -XX:+AlwaysPreTouch"
export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS -verbose:gc -Xloggc:/var/log/hadoop/hdfs-datanode-gc.log"
{noformat}

*hbase-env.sh*
{noformat}
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+UseG1GC"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintGCDetails"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintGCDateStamps"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintGCTimeStamps"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintAdaptiveSizePolicy"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintReferenceGC"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+ParallelRefProcEnabled"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+TieredCompilation"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:-ResizePLAB"

export HBASE_MASTER_OPTS="\$HBASE_MASTER_OPTS -Xms1g -Xmx1g"
export HBASE_MASTER_OPTS="\$HBASE_MASTER_OPTS \$COMMON_HBASE_OPTS"
export HBASE_MASTER_OPTS="\$HBASE_MASTER_OPTS -verbose:gc -Xloggc:/var/log/hbase/hbase-master-gc.log"

export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -Xms32g -Xmx32g"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS \$COMMON_HBASE_OPTS"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -XX:MaxGCPauseMillis=50"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -XX:+UseCondCardMark"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -XX:+AlwaysPreTouch"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -verbose:gc -Xloggc:/var/log/hbase/hbase-regionserver-gc.log"

export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS -Xms1g -Xmx1g"
export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS \$COMMON_HBASE_OPTS"
export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS -XX:+AlwaysPreTouch"
export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS -verbose:gc -Xloggc:/var/log/hbase/hbase-zookeeper-gc.log"
{noformat}

That would handle all of the JVM flag wishlist I think.

bq. ./bin/build_cluster apache_hbase supports a flag that specifies a Java tarball to use

Cool, I will try it.

bq.  I imagine I'd need to modify the code a little bit to handle non-Oracle releases, though.

Maybe not? The OpenJDK based JVM tarballs I build unpack like Oracle releases and run fine on CentOS 6. They just lack JFR. I don't think you're using that.

> Create Docker container cluster infrastructure to enable better testing
> -----------------------------------------------------------------------
>
>                 Key: HBASE-12721
>                 URL: https://issues.apache.org/jira/browse/HBASE-12721
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Dima Spivak
>            Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support as "hbase_docker;" all this did was stand up a standalone cluster from source and start a shell. Now seems like a good time to extend this to be useful for applications that could actual benefit the community, especially around testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up distributed HBase clusters on a single host machine in a couple minutes and run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up an HBase cluster can be done once and then different client source/binary permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any upstream automation on build.apache.org, in part because it's a pain to set up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own top-level module ("hbase-docker" would conveniently fit the existing schema :-)), strives to create a simple framework for deploying "distributed," multi-container Apache HBase clusters.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)