hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4415) Add configuration script for setup HBase (hbase-setup-conf.sh)
Date Sun, 18 Sep 2011 22:30:09 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107551#comment-13107551
] 

Eric Yang commented on HBASE-4415:
----------------------------------

*share/hbase/templates/conf* is an newly added template directory.  People can still use the
default files in conf directory for backward compatibility until hbase-setup-conf.sh become
the standard method of configuring HBase cluster.  For now, there are two copies of hbase-env.sh,
one in conf directory, and one in src/packages/templates/conf.  The copy in src/packages/templates/conf
is designed to be consumed by hbase-setup-conf.sh, and it mimics the changes that happens
to trunk/conf/hbase-env.sh.  If the community agrees that we can use one copy of base-env.sh
and hbase-site.xml for both purpose of script automation and hand edit config file.  I am
more than willing to unify them as one copy and place in src/packages/templates/conf for clarity
to indicate this is for script automation.

Standard Maven practice uses the following structure:

{noformat}
src/main - application code
src/test - test code
{noformat}

In Hadoop related projects, release packaging has introduced new directory structure:

{noformat}
src/packages/tarball - tarball manifest file or related meta data
src/packages/rpm - rpm related meta data
src/packages/deb - debian related meta data
src/packages/template/conf - Configuration template files
{noformat}

This practice is to improve the release packages.  In the past, Hadoop related projects are
releasing a binary/source hybrid tar ball.  This has proven problematic to maintain for different
architecture.  The intention is to improve the system so that we can create binary release
for each architecture and decouple source and binary from tar ball releases.  There are some
more work to be done for packaging separation, but it is not in scope for this jira.

The script is designed to run as root.  For setting up a hbase cluster with hadoop 0.20.205.
 There are a few assumption made for kerberos key tabs.  The key tab file names should be:

namenode          - nn.service.keytab
secondarynamenode - sn.service.keytab
datanode          - dn.service.keytab
jobtracker        - jt.service.keytab
tasktracker       - tt.service.keytab
hbase             - hbase.service.keytab

Compose a shell script that invokes each component's installation script and run them through
pdsh, or use Apache Ambari (HMS):

{noformat}
export namenode=nn1
export datanode=dn1,dn2,dn3,dn4
export secondary_namenode=sn1
export jobtracker=jt1
export tasktracker=tt1,tt2,tt3,tt4
export hbase_regionservers=rs1,rs2,rs3,rs4
export hbase_master=hbm

# Setup Hadoop configuration
/home/hms/apps/hadoop-0.20.205.0/sbin/hadoop-setup-conf.sh \
  --java-home=/home/hms/apps/jdk1.6.0_27 \
  --namenode-host=${namenode} \
  --secondarynamenode-host=${secondary_namenode} \
  --jobtracker-host=${jobtracker} \
  --conf-dir=/etc/hadoop \
  --hdfs-dir=/data/0/hadoop/var/hdfs,/data/1/hadoop/var/hdfs,/data/2/hadoop/var/hdfs,/data/3/hadoop/var/hdfs
\
  --namenode-dir=/data/2/hadoop/var/hdfs/name \
  --mapred-dir=/data/0/tmp/mapred-local,/data/1/tmp/mapred-local,/data/2/tmp/mapred-local,/data/3/tmp/mapred-local
\
  --datanode-dir=/data/0/hadoop/var/hdfs/data,/data/1/hadoop/var/hdfs/data,/data/2/hadoop/var/hdfs/data,/data/3/hadoop/var/hdfs/data
\
  --log-dir=/home/hms/apps/hadoop-0.20.205.0/logs \
  --pid-dir=/home/hms/apps/hadoop-0.20.205.0/logs \
  --hdfs-user=hdfs \
  --mapreduce-user=mr \
  --kerberos-realm=APACHE.ORG \
  --hdfs-user-keytab=/home/hdfs/hdfs.headless.keytab \
  --mapreduce-user-keytab=/home/mr/mr.headless.keytab \
  --datanodes=${datanode} \
  --tasktrackers=${tasktracker} \
  --group=hadoop \
  --auto

# Format Hadoop HDFS
/home/hms/apps/hadoop-0.20.205.0/sbin/hadoop-setup-hdfs.sh \
  --config /etc/hadoop \
  --hdfs-user=hrt_hdfs \
  --mapreduce-user=hrt_mr \
  --kerberos-realm=APACHE.ORG \
  --hdfs-user-keytab=/home/hdfs/hdfs.headless.keytab \
  --format

# Start up secondary name node, datanodes, job tracker, task tracker through hadoop-daemon.sh

# Setup HBase
/home/hms/apps/hbase-0.90.3/sbin/hbase-setup-conf.sh \
  --hadoop-conf=/etc/hadoop \
  --hadoop-home=/home/hms/apps/hadoop-0.20.205.0 \
  --hadoop-namenode=${namenode} \
  --hbase-home=/home/hms/apps/hbase-0.90.3 \
  --hbase-conf=/home/hms/apps/hbase-0.90.3/conf \
  --hbase-log=/home/hms/apps/hbase-0.90.3/logs \
  --hbase-pid=/home/hms/apps/hbase-0.90.3/var/run \
  --java-home=/usr/java/default \
  --kerberos-realm=APACHE.ORG \
  --kerberos-principal-id=hbase/_HOST \
  --keytab-dir=/etc/security/keytabs \
  --regionservers=${hbase_regionservers} \
  --zookeeper-quorum=${hbase_master} \
  --zookeeper-snapshot=/home/hms/apps/var/lib/zookeeper
  --hbase-user=hbase
  --hadoop-replication=3

# Start HBase Master, region servers using hbase-daemon.sh.
{noformat}

Hope this helps.

> Add configuration script for setup HBase (hbase-setup-conf.sh)
> --------------------------------------------------------------
>
>                 Key: HBASE-4415
>                 URL: https://issues.apache.org/jira/browse/HBASE-4415
>             Project: HBase
>          Issue Type: New Feature
>          Components: scripts
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HBASE-4415-1.patch, HBASE-4415.patch
>
>
> The goal of this jura is to provide a installation script for configuring HBase environment
and configuration.  By using the same pattern of *-setup-conf.sh for all Hadoop related projects.
 For HBase, the usage of the script looks like this:
> {noformat}
> usage: ./hbase-setup-conf.sh <parameters>
>   Optional parameters:
>     --hadoop-conf=/etc/hadoop            Set Hadoop configuration directory location
>     --hadoop-home=/usr                       Set Hadoop directory location
>     --hadoop-namenode=localhost              Set Hadoop namenode hostname
>     --hadoop-replication=3                   Set HDFS replication
>     --hbase-home=/usr                        Set HBase directory location
>     --hbase-conf=/etc/hbase                  Set HBase configuration directory location
>     --hbase-log=/var/log/hbase               Set HBase log directory location
>     --hbase-pid=/var/run/hbase               Set HBase pid directory location
>     --hbase-user=hbase                       Set HBase user
>     --java-home=/usr/java/default            Set JAVA_HOME directory location
>     --kerberos-realm=KERBEROS.EXAMPLE.COM    Set Kerberos realm
>     --kerberos-principal-id=_HOST            Set Kerberos principal ID 
>     --keytab-dir=/etc/security/keytabs       Set keytab directory
>     --regionservers=localhost                Set regionservers hostnames
>     --zookeeper-home=/usr                    Set ZooKeeper directory location
>     --zookeeper-quorum=localhost             Set ZooKeeper Quorum
>     --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message