hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
Date Fri, 03 Jun 2016 00:08:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313360#comment-15313360
] 

Andrew Purtell commented on HBASE-12721:
----------------------------------------

bq. clusterdock_run ./bin/build_cluster apache_hbase --hbase-version=0.98 --hbase-git-commit=0.98

Let me try this now. Really liking the ability to specify any git ref. 

bq. Once those images exist, you can pass the location of an .ini-like file as an argument
to start_cluster and it will set configurations for you before starting services 

Ok, what I would like to do is control how hadoop-env.sh and hbase-env.sh are generated. Consider
something like:

*hadoop-env.sh*
{noformat}
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+UseG1GC"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:MaxGCPauseMillis=100"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintGCDetails"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintGCDateStamps"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintGCTimeStamps"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintAdaptiveSizePolicy"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintReferenceGC"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+ParallelRefProcEnabled"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+TieredCompilation"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:-ResizePLAB"

export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS -Xms1g -Xmx1g"
export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS \$COMMON_HDFS_OPTS"
export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS -XX:+AlwaysPreTouch"
export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS -verbose:gc -Xloggc:/var/log/hadoop/hdfs-namenode-gc.log"

export HADOOP_SECONDARYNAMENODE_OPTS="\$HADOOP_SECONDARYNAMENODE_OPTS -Xms1g -Xmx1g"
export HADOOP_SECONDARYNAMENODE_OPTS="\$HADOOP_SECONDARYNAMENODE_OPTS \$COMMON_HDFS_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="\$HADOOP_SECONDARYNAMENODE_OPTS -verbose:gc -Xloggc:/var/log/hadoop/hdfs-secondarynamenode-gc.log"

export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS -Xms1g -Xmx1g"
export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS \$COMMON_HDFS_OPTS"
export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS -XX:+AlwaysPreTouch"
export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS -verbose:gc -Xloggc:/var/log/hadoop/hdfs-datanode-gc.log"
{noformat}

*hbase-env.sh*
{noformat}
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+UseG1GC"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintGCDetails"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintGCDateStamps"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintGCTimeStamps"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintAdaptiveSizePolicy"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintReferenceGC"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+ParallelRefProcEnabled"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+TieredCompilation"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:-ResizePLAB"

export HBASE_MASTER_OPTS="\$HBASE_MASTER_OPTS -Xms1g -Xmx1g"
export HBASE_MASTER_OPTS="\$HBASE_MASTER_OPTS \$COMMON_HBASE_OPTS"
export HBASE_MASTER_OPTS="\$HBASE_MASTER_OPTS -verbose:gc -Xloggc:/var/log/hbase/hbase-master-gc.log"

export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -Xms32g -Xmx32g"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS \$COMMON_HBASE_OPTS"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -XX:MaxGCPauseMillis=50"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -XX:+UseCondCardMark"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -XX:+AlwaysPreTouch"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -verbose:gc -Xloggc:/var/log/hbase/hbase-regionserver-gc.log"

export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS -Xms1g -Xmx1g"
export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS \$COMMON_HBASE_OPTS"
export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS -XX:+AlwaysPreTouch"
export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS -verbose:gc -Xloggc:/var/log/hbase/hbase-zookeeper-gc.log"
{noformat}

That would handle all of the JVM flag wishlist I think.

bq. ./bin/build_cluster apache_hbase supports a flag that specifies a Java tarball to use

Cool, I will try it.

bq.  I imagine I'd need to modify the code a little bit to handle non-Oracle releases, though.

Maybe not? The OpenJDK based JVM tarballs I build unpack like Oracle releases and run fine
on CentOS 6. They just lack JFR. I don't think you're using that.

> Create Docker container cluster infrastructure to enable better testing
> -----------------------------------------------------------------------
>
>                 Key: HBASE-12721
>                 URL: https://issues.apache.org/jira/browse/HBASE-12721
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Dima Spivak
>            Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support as "hbase_docker;"
all this did was stand up a standalone cluster from source and start a shell. Now seems like
a good time to extend this to be useful for applications that could actual benefit the community,
especially around testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up distributed
HBase clusters on a single host machine in a couple minutes and run our awesome hbase-it suite
against it.
> - Binary compatibility testing of an HBase client is easiest when standing up an HBase
cluster can be done once and then different client source/binary permutations run against
it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any upstream
automation on build.apache.org, in part because it's a pain to set up x-node clusters on Apache
infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own top-level
module ("hbase-docker" would conveniently fit the existing schema :-)), strives to create
a simple framework for deploying "distributed," multi-container Apache HBase clusters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message