Apache HBase Configuration

- - Apache HBase Configuration - This chapter expands upon the chapter to further explain - configuration of Apache HBase. Please read this chapter carefully, especially to ensure that your HBase testing and deployment goes - smoothly, and prevent data loss. - - Apache HBase uses the same configuration system as Apache Hadoop. All configuration files - are located in the conf/ directory, which needs to be kept in sync for each - node on your cluster. - - - HBase Configuration Files - - backup-masters - - Not present by default. A plain-text file which lists hosts on which the Master should - start a backup Master process, one host per line. - - - - hadoop-metrics2-hbase.properties - - Used to connect HBase Hadoop's Metrics2 framework. See the Hadoop Wiki - entry for more information on Metrics2. Contains only commented-out examples by - default. - - - - hbase-env.cmd and hbase-env.sh - - Script for Windows and Linux / Unix environments to set up the working environment for - HBase, including the location of Java, Java options, and other environment variables. The - file contains many commented-out examples to provide guidance. - - In HBase 0.98.5 and newer, you must set JAVA_HOME on each node of - your cluster. hbase-env.sh provides a handy mechanism to do - this. - - - - - hbase-policy.xml - - The default policy configuration file used by RPC servers to make authorization - decisions on client requests. Only used if HBase security () is enabled. - - - - hbase-site.xml - - The main HBase configuration file. This file specifies configuration options which - override HBase's default configuration. You can view (but do not edit) the default - configuration file at docs/hbase-default.xml. You can also view the - entire effective configuration for your cluster (defaults and overrides) in the - HBase Configuration tab of the HBase Web UI. - - - - log4j.properties - - Configuration file for HBase logging via log4j. - - - - regionservers - - A plain-text file containing a list of hosts which should run a RegionServer in your - HBase cluster. By default this file contains the single entry - localhost. It should contain a list of hostnames or IP addresses, one - per line, and should only contain localhost if each node in your - cluster will run a RegionServer on its localhost interface. - - - - - - Checking XML Validity - When you edit XML, it is a good idea to use an XML-aware editor to be sure that your - syntax is correct and your XML is well-formed. You can also use the xmllint - utility to check that your XML is well-formed. By default, xmllint re-flows - and prints the XML to standard output. To check for well-formedness and only print output if - errors exist, use the command xmllint -noout - filename.xml. - - - - Keep Configuration In Sync Across the Cluster - When running in distributed mode, after you make an edit to an HBase configuration, make - sure you copy the content of the conf/ directory to all nodes of the - cluster. HBase will not do this for you. Use rsync, scp, - or another secure mechanism for copying the configuration files to your nodes. For most - configuration, a restart is needed for servers to pick up changes An exception is dynamic - configuration. to be described later below. - - -

- Basic Prerequisites - This section lists required services and some required system configuration. - - - Java - - HBase requires at least Java 6 from Oracle. The following table lists - which JDK version are compatible with each version of HBase. - - - - - HBase Version - JDK 6 - JDK 7 - JDK 8 - - - - - 1.0 - Not Supported - yes - Running with JDK 8 will work but is not well tested. - - - 0.98 - yes - yes - Running with JDK 8 works but is not well tested. Building with JDK 8 would - require removal of the deprecated remove() method of the PoolMap class and is under - consideration. See ee HBASE-7608 - for more information about JDK 8 support. - - - 0.96 - yes - yes - - - - 0.94 - yes - yes - - - - -

- - - In HBase 0.98.5 and newer, you must set JAVA_HOME on each node of - your cluster. hbase-env.sh provides a handy mechanism to do - this. - - - - Operating System Utilities - - ssh - - HBase uses the Secure Shell (ssh) command and utilities extensively to communicate - between cluster nodes. Each server in the cluster must be running ssh - so that the Hadoop and HBase daemons can be managed. You must be able to connect to all - nodes via SSH, including the local node, from the Master as well as any backup Master, - using a shared key rather than a password. You can see the basic methodology for such a - set-up in Linux or Unix systems at . If your cluster nodes use OS X, see the - section, SSH: - Setting up Remote Desktop and Enabling Self-Login on the Hadoop wiki. - - - - DNS - - HBase uses the local hostname to self-report its IP address. Both forward and - reverse DNS resolving must work in versions of HBase previous to 0.92.0. The hadoop-dns-checker - tool can be used to verify DNS is working correctly on the cluster. The project - README file provides detailed instructions on usage. - - If your server has multiple network interfaces, HBase defaults to using the - interface that the primary hostname resolves to. To override this behavior, set the - hbase.regionserver.dns.interface property to a different interface. This - will only work if each server in your cluster uses the same network interface - configuration. - - To choose a different DNS nameserver than the system default, set the - hbase.regionserver.dns.nameserver property to the IP address of - that nameserver. - - - - Loopback IP - - Prior to hbase-0.96.0, HBase only used the IP address - 127.0.0.1 to refer to localhost, and this could - not be configured. See . - - - - NTP - - The clocks on cluster nodes should be synchronized. A small amount of variation is - acceptable, but larger amounts of skew can cause erratic and unexpected behavior. Time - synchronization is one of the first things to check if you see unexplained problems in - your cluster. It is recommended that you run a Network Time Protocol (NTP) service, or - another time-synchronization mechanism, on your cluster, and that all nodes look to the - same service for time synchronization. See the Basic NTP - Configuration at The Linux Documentation Project (TLDP) - to set up NTP. - - - - - Limits on Number of Files and Processes (ulimit) - - ulimit - - nproc - - - - - Apache HBase is a database. It requires the ability to open a large number of files - at once. Many Linux distributions limit the number of files a single user is allowed to - open to 1024 (or 256 on older versions of OS X). - You can check this limit on your servers by running the command ulimit - -n when logged in as the user which runs HBase. See for some of the problems you may - experience if the limit is too low. You may also notice errors such as the - following: - -2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException -2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901 - - It is recommended to raise the ulimit to at least 10,000, but more likely 10,240, - because the value is usually expressed in multiples of 1024. Each ColumnFamily has at - least one StoreFile, and possibly more than 6 StoreFiles if the region is under load. - The number of open files required depends upon the number of ColumnFamilies and the - number of regions. The following is a rough formula for calculating the potential number - of open files on a RegionServer. - - Calculate the Potential Number of Open Files - (StoreFiles per ColumnFamily) x (regions per RegionServer) - - For example, assuming that a schema had 3 ColumnFamilies per region with an average - of 3 StoreFiles per ColumnFamily, and there are 100 regions per RegionServer, the JVM - will open 3 * 3 * 100 = 900 file descriptors, not counting open JAR files, configuration - files, and others. Opening a file does not take many resources, and the risk of allowing - a user to open too many files is minimal. - Another related setting is the number of processes a user is allowed to run at once. - In Linux and Unix, the number of processes is set using the ulimit -u - command. This should not be confused with the nproc command, which - controls the number of CPUs available to a given user. Under load, a - nproc that is too low can cause OutOfMemoryError exceptions. See - Jack Levin's major - hdfs issues thread on the hbase-users mailing list, from 2011. - Configuring the fmaximum number of ile descriptors and processes for the user who is - running the HBase process is an operating system configuration, rather than an HBase - configuration. It is also important to be sure that the settings are changed for the - user that actually runs HBase. To see which user started HBase, and that user's ulimit - configuration, look at the first line of the HBase log for that instance. A useful read - setting config on you hadoop cluster is Aaron Kimballs' Configuration Parameters: What can you just ignore? - - <command>ulimit</command> Settings on Ubuntu - To configure ulimit settings on Ubuntu, edit - /etc/security/limits.conf, which is a space-delimited file with - four columns. Refer to the man - page for limits.conf for details about the format of this file. In the - following example, the first line sets both soft and hard limits for the number of - open files (nofile) to 32768 for the operating - system user with the username hadoop. The second line sets the - number of processes to 32000 for the same user. - - -hadoop - nofile 32768 -hadoop - nproc 32000 - - The settings are only applied if the Pluggable Authentication Module (PAM) - environment is directed to use them. To configure PAM to use these limits, be sure that - the /etc/pam.d/common-session file contains the following line: - session required pam_limits.so - - - - - Windows - - - Prior to HBase 0.96, testing for running HBase on Microsoft Windows was limited. - Running a on Windows nodes is not recommended for production systems. - - To run versions of HBase prior to 0.96 on Microsoft Windows, you must install Cygwin and run HBase within the Cygwin - environment. This provides support for Linux/Unix commands and scripts. The full details are explained in the Windows Installation guide. Also search - our user mailing list to pick up latest fixes figured by Windows users. - Post-hbase-0.96.0, hbase runs natively on windows with supporting - *.cmd scripts bundled. - - - - - -

- <link - xlink:href="http://hadoop.apache.org">Hadoop</link><indexterm> - <primary>Hadoop</primary> - </indexterm> - The following table summarizes the versions of Hadoop supported with each version of - HBase. Based on the version of HBase, you should select the most - appropriate version of Hadoop. You can use Apache Hadoop, or a vendor's distribution of - Hadoop. No distinction is made here. See - for information about vendors of Hadoop. - - Hadoop 2.x is recommended. - Hadoop 2.x is faster and includes features, such as short-circuit reads, which will - help improve your HBase random read profile. Hadoop 2.x also includes important bug fixes - that will improve your overall HBase experience. HBase 0.98 drops support for Hadoop 1.0, deprecates use of Hadoop 1.1+, - and HBase 1.0 will not support Hadoop 1.x. - - Use the following legend to interpret this table: - - S = supported and tested, - X = not supported, - NT = it should run, but not tested enough. - - - - Hadoop version support matrix - - - - - - - - - - - HBase-0.92.x - HBase-0.94.x - HBase-0.96.x - HBase-0.98.x (Support for Hadoop 1.1+ is deprecated.) - HBase-1.0.x (Hadoop 1.x is NOT supported) - - - - - Hadoop-0.20.205 - S - X - X - X - X - - - Hadoop-0.22.x - S - X - X - X - X - - - Hadoop-1.0.x - X - X - X - X - X - - - Hadoop-1.1.x - NT - S - S - NT - X - - - Hadoop-0.23.x - X - S - NT - X - X - - - Hadoop-2.0.x-alpha - X - NT - X - X - X - - - Hadoop-2.1.0-beta - X - NT - S - X - X - - - Hadoop-2.2.0 - X - NT - S - S - NT - - - Hadoop-2.3.x - X - NT - S - S - NT - - - Hadoop-2.4.x - X - NT - S - S - S - - - Hadoop-2.5.x - X - NT - S - S - S - - - - -

- - - Replace the Hadoop Bundled With HBase! - Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its - lib directory. The bundled jar is ONLY for use in standalone mode. - In distributed mode, it is critical that the version of Hadoop that - is out on your cluster match what is under HBase. Replace the hadoop jar found in the - HBase lib directory with the hadoop jar you are running on your cluster to avoid version - mismatch issues. Make sure you replace the jar in HBase everywhere on your cluster. Hadoop - version mismatch issues have various manifestations but often all looks like its hung up. - - -

- Apache HBase 0.94 with Hadoop 2 - To get 0.94.x to run on hadoop 2.2.0, you need to change the hadoop - 2 and protobuf versions in the pom.xml: Here is a diff with - pom.xml changes: - 1.4.3 - 1.2.16 - 1.8.5 -- 2.4.0a -+ 2.5.0 - 1.0.1 - 0.8.0 - 3.4.5 -@@ -2241,7 +2241,7 @@ - - - -- 2.0.0-alpha -+ 2.2.0 - 1.6.1 - - ]]> - - The next step is to regenerate Protobuf files and assuming that the Protobuf - has been installed: - - - Go to the hbase root folder, using the command line; - - - Type the following commands: - - - - - - - - - Building against the hadoop 2 profile by running something like the - following command: - $ mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests -

- Apache HBase 0.92 and 0.94 - HBase 0.92 and 0.94 versions can work with Hadoop versions, 0.20.205, 0.22.x, 1.0.x, - and 1.1.x. HBase-0.94 can additionally work with Hadoop-0.23.x and 2.x, but you may have - to recompile the code using the specific maven profile (see top level pom.xml) -

- -

- Apache HBase 0.96 - As of Apache HBase 0.96.x, Apache Hadoop 1.0.x at least is required. Hadoop 2 is - strongly encouraged (faster but also has fixes that help MTTR). We will no longer run - properly on older Hadoops such as 0.20.205 or branch-0.20-append. Do not move to Apache - HBase 0.96.x if you cannot upgrade your Hadoop.. See HBase, mail # dev - DISCUSS: - Have hbase require at least hadoop 1.0.0 in hbase 0.96.0? -

- -

- Hadoop versions 0.20.x - 1.x - HBase will lose data unless it is running on an HDFS that has a durable - sync implementation. DO NOT use Hadoop 0.20.2, Hadoop 0.20.203.0, and - Hadoop 0.20.204.0 which DO NOT have this attribute. Currently only Hadoop versions - 0.20.205.x or any release in excess of this version -- this includes hadoop-1.0.0 -- have - a working, durable sync. The Cloudera blog post An - update on Apache Hadoop 1.0 by Charles Zedlweski has a nice exposition on how all - the Hadoop versions relate. Its worth checking out if you are having trouble making sense - of the Hadoop version morass. - Sync has to be explicitly enabled by setting - dfs.support.append equal to true on both the client side -- in - hbase-site.xml -- and on the serverside in - hdfs-site.xml (The sync facility HBase needs is a subset of the - append code path). - - dfs.support.append - true -]]> - You will have to restart your cluster after making this edit. Ignore the - chicken-little comment you'll find in the hdfs-default.xml in the - description for the dfs.support.append configuration. -

- Apache HBase on Secure Hadoop - Apache HBase will run on any Hadoop 0.20.x that incorporates Hadoop security features - as long as you do as suggested above and replace the Hadoop jar that ships with HBase with - the secure version. If you want to read more about how to setup Secure HBase, see . -

- -

- <varname>dfs.datanode.max.transfer.threads</varname><indexterm> - <primary>dfs.datanode.max.transfer.threads</primary> - </indexterm> - - An HDFS datanode has an upper bound on the number of files that it will serve - at any one time. Before doing any loading, make sure you have configured - Hadoop's conf/hdfs-site.xml, setting the - dfs.datanode.max.transfer.threads value to at least the following: - - - dfs.datanode.max.transfer.threads - 4096 - - ]]> - - Be sure to restart your HDFS after making the above configuration. - - Not having this configuration in place makes for strange-looking failures. One - manifestation is a complaint about missing blocks. For example: - 10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block - blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes - contain current block. Will get new block locations from namenode and retry... - See also and note that this - property was previously known as dfs.datanode.max.xcievers (e.g. - - Hadoop HDFS: Deceived by Xciever). - - - -

- -

- ZooKeeper Requirements - ZooKeeper 3.4.x is required as of HBase 1.0.0. HBase makes use of the - multi functionality that is only available since 3.4.0 - (The useMulti is defaulted true in HBase 1.0.0). - See HBASE-12241 The crash of regionServer when taking deadserver's replication queue breaks replication - and Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix for background. -

- -

- HBase run modes: Standalone and Distributed - - HBase has two run modes: and . Out of the box, HBase runs in standalone mode. Whatever your mode, - you will need to configure HBase by editing files in the HBase conf - directory. At a minimum, you must edit conf/hbase-env.sh to tell HBase which - java to use. In this file you set HBase environment variables such as the - heapsize and other options for the JVM, the preferred location for - log files, etc. Set JAVA_HOME to point at the root of your - java install. - -

- Standalone HBase - - This is the default mode. Standalone mode is what is described in the section. In standalone mode, HBase does not use HDFS -- it uses - the local filesystem instead -- and it runs all HBase daemons and a local ZooKeeper all up - in the same JVM. Zookeeper binds to a well known port so clients may talk to HBase. -

- -

- Distributed - - Distributed mode can be subdivided into distributed but all daemons run on a single node - -- a.k.a pseudo-distributed-- and - fully-distributed where the daemons are spread across all nodes in - the cluster. The pseudo-distributed vs fully-distributed nomenclature comes from Hadoop. - - Pseudo-distributed mode can run against the local filesystem or it can run against an - instance of the Hadoop Distributed File System (HDFS). - Fully-distributed mode can ONLY run on HDFS. See the Hadoop - requirements and instructions for how to set up HDFS for Hadoop 1.x. A good - walk-through for setting up HDFS on Hadoop 2 is at http://www.alexjf.net/blog/distributed-systems/hadoop-yarn-installation-definitive-guide. - - Below we describe the different distributed setups. Starting, verification and - exploration of your install, whether a pseudo-distributed or - fully-distributed configuration is described in a section that - follows, . The same verification script applies to both deploy types. -

- Pseudo-distributed - - Pseudo-Distributed Quickstart - A quickstart has been added to the chapter. See . Some of the information that was originally in this - section has been moved there. - - - A pseudo-distributed mode is simply a fully-distributed mode run on a single host. Use - this configuration testing and prototyping on HBase. Do not use this configuration for - production nor for evaluating HBase performance. - -

- -

- Fully-distributed - By default, HBase runs in standalone mode. Both standalone mode and pseudo-distributed - mode are provided for the purposes of small-scale testing. For a production environment, - distributed mode is appropriate. In distributed mode, multiple instances of HBase daemons - run on multiple servers in the cluster. - Just as in pseudo-distributed mode, a fully distributed configuration requires that you - set the hbase-cluster.distributed property to true. - Typically, the hbase.rootdir is configured to point to a highly-available HDFS - filesystem. - In addition, the cluster is configured so that multiple cluster nodes enlist as - RegionServers, ZooKeeper QuorumPeers, and backup HMaster servers. These configuration basics - are all demonstrated in . - - - Distributed RegionServers - Typically, your cluster will contain multiple RegionServers all running on different - servers, as well as primary and backup Master and Zookeeper daemons. The - conf/regionservers file on the master server contains a list of - hosts whose RegionServers are associated with this cluster. Each host is on a separate - line. All hosts listed in this file will have their RegionServer processes started and - stopped when the master server starts or stops. - - - - ZooKeeper and HBase - See section for ZooKeeper setup for HBase. - - - - Example Distributed HBase Cluster - This is a bare-bones conf/hbase-site.xml for a distributed HBase - cluster. A cluster that is used for real-world work would contain more custom - configuration parameters. Most HBase configuration directives have default values, which - are used unless the value is overridden in the hbase-site.xml. See for more information. - - - hbase.rootdir - hdfs://namenode.example.org:8020/hbase - - - hbase.cluster.distributed - true - - - hbase.zookeeper.quorum - node-a.example.com,node-b.example.com,node-c.example.com - - -]]> - - This is an example conf/regionservers file, which contains a list - of each node that should run a RegionServer in the cluster. These nodes need HBase - installed and they need to use the same contents of the conf/ - directory as the Master server.. - -node-a.example.com -node-b.example.com -node-c.example.com - - This is an example conf/backup-masters file, which contains a - list of each node that should run a backup Master instance. The backup Master instances - will sit idle unless the main Master becomes unavailable. - -node-b.example.com -node-c.example.com - - - - Distributed HBase Quickstart - See for a walk-through of a simple three-node - cluster configuration with multiple ZooKeeper, backup HMaster, and RegionServer - instances. - - - - HDFS Client Configuration - - Of note, if you have made HDFS client configuration on your Hadoop cluster, such as - configuration directives for HDFS clients, as opposed to server-side configurations, you - must use one of the following methods to enable HBase to see and use these configuration - changes: - - - Add a pointer to your HADOOP_CONF_DIR to the - HBASE_CLASSPATH environment variable in - hbase-env.sh. - - - - Add a copy of hdfs-site.xml (or - hadoop-site.xml) or, better, symlinks, under - ${HBASE_HOME}/conf, or - - - - if only a small set of HDFS client configurations, add them to - hbase-site.xml. - - - - - An example of such an HDFS client configuration is dfs.replication. - If for example, you want to run with a replication factor of 5, hbase will create files with - the default of 3 unless you do the above to make the configuration available to - HBase. -

- -

- Running and Confirming Your Installation - - - - Make sure HDFS is running first. Start and stop the Hadoop HDFS daemons by running - bin/start-hdfs.sh over in the HADOOP_HOME - directory. You can ensure it started properly by testing the put and - get of files into the Hadoop filesystem. HBase does not normally use - the mapreduce daemons. These do not need to be started. - If you are managing your own ZooKeeper, start it and confirm its - running else, HBase will start up ZooKeeper for you as part of its start process. - Start HBase with the following command: - bin/start-hbase.sh - Run the above from the HBASE_HOME directory. - You should now have a running HBase instance. HBase logs can be found in the - logs subdirectory. Check them out especially if HBase had trouble - starting. - - HBase also puts up a UI listing vital attributes. By default its deployed on the Master - host at port 16010 (HBase RegionServers listen on port 16020 by default and put up an - informational http server at 16030). If the Master were running on a host named - master.example.org on the default port, to see the Master's homepage - you'd point your browser at http://master.example.org:16010. - - Prior to HBase 0.98, the default ports the master ui was deployed on port 16010, and the - HBase RegionServers would listen on port 16020 by default and put up an informational http - server at 16030. - - Once HBase has started, see the for how to create tables, add data, scan your insertions, and - finally disable and drop your tables. - - To stop HBase after exiting the HBase shell enter - $ ./bin/stop-hbase.sh -stopping hbase............... - Shutdown can take a moment to complete. It can take longer if your cluster is comprised - of many machines. If you are running a distributed operation, be sure to wait until HBase - has shut down completely before stopping the Hadoop daemons. -

- - - - - -

- Configuration Files - -

- <filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename> - Just as in Hadoop where you add site-specific HDFS configuration to the - hdfs-site.xml file, for HBase, site specific customizations go into - the file conf/hbase-site.xml. For the list of configurable properties, - see below or view the raw - hbase-default.xml source file in the HBase source code at - src/main/resources. - Not all configuration options make it out to hbase-default.xml. - Configuration that it is thought rare anyone would change can exist only in code; the only - way to turn up such configurations is via a reading of the source code itself. - Currently, changes here will require a cluster restart for HBase to notice the change. - - - - -

- - <para> - <emphasis>This file is fallback content</emphasis>. If you are seeing this, something - is wrong with the build of the HBase documentation or you are doing pre-build - verification. </para> - <para> The file hbase-default.xml is generated as part of the build of the hbase site. - See the hbase <filename>pom.xml</filename>. The generated file is a docbook glossary. </para> - <section> - <title>IDs that are auto-generated and cause validation errors if not present - Each of these is a reference to a configuration file parameter which will cause - an error if you are using the fallback content here. This is a dirty dirty hack. -

- fail.fast.expired.active.master - -

- "hbase.hregion.memstore.flush.size" - -

- hbase.hstore.bytes.per.checksum - -

- hbase.online.schema.update.enable - -

- hbase.regionserver.global.memstore.size - -

- hbase.hregion.max.filesize - -

- hbase.hstore.BlockingStoreFiles - -

- hfile.block.cache.size - -

- copy.table - -

- hbase.hstore.checksum.algorithm - -

- hbase.zookeeper.useMulti - -

- hbase.hregion.memstore.block.multiplier - -

- hbase.regionserver.global.memstore.size.lower.limit - -

- - -

- -

- <filename>hbase-env.sh</filename> - Set HBase environment variables in this file. Examples include options to pass the JVM - on start of an HBase daemon such as heap size and garbage collector configs. You can also - set configurations for HBase configuration, log directories, niceness, ssh options, where to - locate process pid files, etc. Open the file at conf/hbase-env.sh and - peruse its content. Each option is fairly well documented. Add your own environment - variables here if you want them read by HBase daemons on startup. - Changes here will require a cluster restart for HBase to notice the change. -

- -

- <filename>log4j.properties</filename> - Edit this file to change rate at which HBase files are rolled and to change the level at - which HBase logs messages. - Changes here will require a cluster restart for HBase to notice the change though log - levels can be changed for particular daemons via the HBase UI. -

- -

- Client configuration and dependencies connecting to an HBase cluster - If you are running HBase in standalone mode, you don't need to configure anything for - your client to work provided that they are all on the same machine. - Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for - current critical locations. ZooKeeper is where all these values are kept. Thus clients - require the location of the ZooKeeper ensemble information before they can do anything else. - Usually this the ensemble location is kept out in the hbase-site.xml - and is picked up by the client from the CLASSPATH. - - If you are configuring an IDE to run a HBase client, you should include the - conf/ directory on your classpath so - hbase-site.xml settings can be found (or add - src/test/resources to pick up the hbase-site.xml used by tests). - Minimally, a client of HBase needs several libraries in its - CLASSPATH when connecting to a cluster, including: - -commons-configuration (commons-configuration-1.6.jar) -commons-lang (commons-lang-2.5.jar) -commons-logging (commons-logging-1.1.1.jar) -hadoop-core (hadoop-core-1.0.0.jar) -hbase (hbase-0.92.0.jar) -log4j (log4j-1.2.16.jar) -slf4j-api (slf4j-api-1.5.8.jar) -slf4j-log4j (slf4j-log4j12-1.5.8.jar) -zookeeper (zookeeper-3.4.2.jar) - - An example basic hbase-site.xml for client only might look as - follows: - - - - hbase.zookeeper.quorum - example1,example2,example3 - The directory shared by region servers. - - - -]]> - - -

- Java client configuration - The configuration used by a Java client is kept in an HBaseConfiguration - instance. The factory method on HBaseConfiguration, - HBaseConfiguration.create();, on invocation, will read in the content of - the first hbase-site.xml found on the client's - CLASSPATH, if one is present (Invocation will also factor in any - hbase-default.xml found; an hbase-default.xml ships inside the - hbase.X.X.X.jar). It is also possible to specify configuration - directly without having to read from a hbase-site.xml. For example, - to set the ZooKeeper ensemble for the cluster programmatically do as follows: - Configuration config = HBaseConfiguration.create(); -config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zookeeper locally - If multiple ZooKeeper instances make up your ZooKeeper ensemble, they may be specified in - a comma-separated list (just as in the hbase-site.xml file). This - populated Configuration instance can then be passed to an HTable, - and so on. -

- - - - -

- Example Configurations - -

- Basic Distributed HBase Install - - Here is an example basic configuration for a distributed ten node cluster. The nodes are - named example0, example1, etc., through node - example9 in this example. The HBase Master and the HDFS namenode are - running on the node example0. RegionServers run on nodes - example1-example9. A 3-node ZooKeeper ensemble runs - on example1, example2, and example3 - on the default ports. ZooKeeper data is persisted to the directory - /export/zookeeper. Below we show what the main configuration files -- - hbase-site.xml, regionservers, and - hbase-env.sh -- found in the HBase conf - directory might look like. - -

- <filename>hbase-site.xml</filename> - - - - - - - hbase.zookeeper.quorum - example1,example2,example3 - The directory shared by RegionServers. - - - - hbase.zookeeper.property.dataDir - /export/zookeeper - Property from ZooKeeper config zoo.cfg. - The directory where the snapshot is stored. - - - - hbase.rootdir - hdfs://example0:8020/hbase - The directory shared by RegionServers. - - - - hbase.cluster.distributed - true - The mode the cluster will be in. Possible values are - false: standalone and pseudo-distributed setups with managed Zookeeper - true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) - - - -]]> - -

- -

- <filename>regionservers</filename> - - In this file you list the nodes that will run RegionServers. In our case, these nodes - are example1-example9. - - -example1 -example2 -example3 -example4 -example5 -example6 -example7 -example8 -example9 - -

- -

- <filename>hbase-env.sh</filename> - - The following lines in the hbase-env.sh file show how to set the - JAVA_HOME environment variable (required for HBase 0.98.5 and newer) and - set the heap to 4 GB (rather than the default value of 1 GB). If you copy and paste this - example, be sure to adjust the JAVA_HOME to suit your environment. - - -# The java implementation to use. -export JAVA_HOME=/usr/java/jdk1.7.0/ - -# The maximum amount of heap to use, in MB. Default is 1000. -export HBASE_HEAPSIZE=4096 - - - Use rsync to copy the content of the conf - directory to all nodes of the cluster. -

- - - -

- The Important Configurations - Below we list what the important Configurations. We've divided this - section into required configuration and worth-a-look recommended configs. - - -

- Required Configurations - Review the and sections. -

- Big Cluster Configurations - If a cluster with a lot of regions, it is possible if an eager beaver regionserver - checks in soon after master start while all the rest in the cluster are laggardly, this - first server to checkin will be assigned all regions. If lots of regions, this first - server could buckle under the load. To prevent the above scenario happening up the - hbase.master.wait.on.regionservers.mintostart from its default value - of 1. See HBASE-6389 Modify the - conditions to ensure that Master waits for sufficient number of Region Servers before - starting region assignments for more detail. -

- If a backup Master, making primary Master fail fast - If the primary Master loses its connection with ZooKeeper, it will fall into a loop - where it keeps trying to reconnect. Disable this functionality if you are running more - than one Master: i.e. a backup Master. Failing to do so, the dying Master may continue to - receive RPCs though another Master has assumed the role of primary. See the configuration . -

- -

- Recommended Configurations -

- ZooKeeper Configuration -

- <varname>zookeeper.session.timeout</varname> - The default timeout is three minutes (specified in milliseconds). This means that if - a server crashes, it will be three minutes before the Master notices the crash and - starts recovery. You might like to tune the timeout down to a minute or even less so the - Master notices failures the sooner. Before changing this value, be sure you have your - JVM garbage collection configuration under control otherwise, a long garbage collection - that lasts beyond the ZooKeeper session timeout will take out your RegionServer (You - might be fine with this -- you probably want recovery to start on the server if a - RegionServer has been in GC for a long period of time). - - To change this configuration, edit hbase-site.xml, copy the - changed file around the cluster and restart. - - We set this value high to save our having to field noob questions up on the mailing - lists asking why a RegionServer went down during a massive import. The usual cause is - that their JVM is untuned and they are running into long GC pauses. Our thinking is that - while users are getting familiar with HBase, we'd save them having to know all of its - intricacies. Later when they've built some confidence, then they can play with - configuration such as this. -

- Number of ZooKeeper Instances - See . -

- HDFS Configurations -

- dfs.datanode.failed.volumes.tolerated - This is the "...number of volumes that are allowed to fail before a datanode stops - offering service. By default any volume failure will cause a datanode to shutdown" from - the hdfs-default.xml description. If you have > three or four - disks, you might want to set this to 1 or if you have many disks, two or more. -

- <varname>hbase.regionserver.handler.count</varname> - This setting defines the number of threads that are kept open to answer incoming - requests to user tables. The rule of thumb is to keep this number low when the payload per - request approaches the MB (big puts, scans using a large cache) and high when the payload - is small (gets, small puts, ICVs, deletes). The total size of the queries in progress is - limited by the setting "hbase.ipc.server.max.callqueue.size". - It is safe to set that number to the maximum number of incoming clients if their - payload is small, the typical example being a cluster that serves a website since puts - aren't typically buffered and most of the operations are gets. - The reason why it is dangerous to keep this setting high is that the aggregate size - of all the puts that are currently happening in a region server may impose too much - pressure on its memory, or even trigger an OutOfMemoryError. A region server running on - low memory will trigger its JVM's garbage collector to run more frequently up to a point - where GC pauses become noticeable (the reason being that all the memory used to keep all - the requests' payloads cannot be trashed, no matter how hard the garbage collector tries). - After some time, the overall cluster throughput is affected since every request that hits - that region server will take longer, which exacerbates the problem even more. - You can get a sense of whether you have too little or too many handlers by on an individual RegionServer then tailing its logs (Queued - requests consume memory). -

- Configuration for large memory machines - HBase ships with a reasonable, conservative configuration that will work on nearly - all machine types that people might want to test with. If you have larger machines -- - HBase has 8G and larger heap -- you might the following configuration options helpful. - TODO. - -

- -

- Compression - You should consider enabling ColumnFamily compression. There are several options that - are near-frictionless and in most all cases boost performance by reducing the size of - StoreFiles and thus reducing I/O. - See for more information. -

- Configuring the size and number of WAL files - HBase uses to recover the memstore data that has not been flushed to disk in case - of an RS failure. These WAL files should be configured to be slightly smaller than HDFS - block (by default, HDFS block is 64Mb and WAL file is ~60Mb). - HBase also has a limit on number of WAL files, designed to ensure there's never too - much data that needs to be replayed during recovery. This limit needs to be set according - to memstore configuration, so that all the necessary data would fit. It is recommended to - allocated enough WAL files to store at least that much data (when all memstores are close - to full). For example, with 16Gb RS heap, default memstore settings (0.4), and default WAL - file size (~60Mb), 16Gb*0.4/60, the starting point for WAL file count is ~109. However, as - all memstores are not expected to be full all the time, less WAL files can be - allocated. -

- Managed Splitting - HBase generally handles splitting your regions, based upon the settings in your - hbase-default.xml and hbase-site.xml - configuration files. Important settings include - hbase.regionserver.region.split.policy, - hbase.hregion.max.filesize, - hbase.regionserver.regionSplitLimit. A simplistic view of splitting - is that when a region grows to hbase.hregion.max.filesize, it is split. - For most use patterns, most of the time, you should use automatic splitting. See for more information about manual region - splitting. - Instead of allowing HBase to split your regions automatically, you can choose to - manage the splitting yourself. This feature was added in HBase 0.90.0. Manually managing - splits works if you know your keyspace well, otherwise let HBase figure where to split for you. - Manual splitting can mitigate region creation and movement under load. It also makes it so - region boundaries are known and invariant (if you disable region splitting). If you use manual - splits, it is easier doing staggered, time-based major compactions spread out your network IO - load. - - - Disable Automatic Splitting - To disable automatic splitting, set hbase.hregion.max.filesize to - a very large value, such as 100 GB It is not recommended to set it to - its absolute maximum value of Long.MAX_VALUE. - - - Automatic Splitting Is Recommended - If you disable automatic splits to diagnose a problem or during a period of fast - data growth, it is recommended to re-enable them when your situation becomes more - stable. The potential benefits of managing region splits yourself are not - undisputed. - - - Determine the Optimal Number of Pre-Split Regions - The optimal number of pre-split regions depends on your application and environment. - A good rule of thumb is to start with 10 pre-split regions per server and watch as data - grows over time. It is better to err on the side of too few regions and perform rolling - splits later. The optimal number of regions depends upon the largest StoreFile in your - region. The size of the largest StoreFile will increase with time if the amount of data - grows. The goal is for the largest region to be just large enough that the compaction - selection algorithm only compacts it during a timed major compaction. Otherwise, the - cluster can be prone to compaction storms where a large number of regions under - compaction at the same time. It is important to understand that the data growth causes - compaction storms, and not the manual split decision. - - If the regions are split into too many large regions, you can increase the major - compaction interval by configuring HConstants.MAJOR_COMPACTION_PERIOD. - HBase 0.90 introduced org.apache.hadoop.hbase.util.RegionSplitter, - which provides a network-IO-safe rolling split of all regions. -

- Managed Compactions - By default, major compactions are scheduled to run once in a 7-day period. Prior to HBase 0.96.x, major - compactions were scheduled to happen once per day by default. - If you need to control exactly when and how often major compaction runs, you can - disable managed major compactions. See the entry for - hbase.hregion.majorcompaction in the table for details. - - Do Not Disable Major Compactions - Major compactions are absolutely necessary for StoreFile clean-up. Do not disable - them altogether. You can run major compactions manually via the HBase shell or via the HBaseAdmin - API. - - For more information about compactions and the compaction file selection process, see -

- -

- Speculative Execution - Speculative Execution of MapReduce tasks is on by default, and for HBase clusters it - is generally advised to turn off Speculative Execution at a system-level unless you need - it for a specific case, where it can be configured per-job. Set the properties - mapreduce.map.speculative and - mapreduce.reduce.speculative to false. -

Other Configurations -

Balancer - The balancer is a periodic operation which is run on the master to redistribute regions on the cluster. It is configured via - hbase.balancer.period and defaults to 300000 (5 minutes). - See for more information on the LoadBalancer. - -

Disabling Blockcache - Do not turn off block cache (You'd do it by setting hbase.block.cache.size to zero). - Currently we do not do well if you do this because the regionserver will spend all its time loading hfile - indices over and over again. If your working set it such that block cache does you no good, at least - size the block cache such that hfile indices will stay up in the cache (you can get a rough idea - on the size you need by surveying regionserver UIs; you'll see index block size accounted near the - top of the webpage). -

- <link xlink:href="http://en.wikipedia.org/wiki/Nagle's_algorithm">Nagle's</link> or the small package problem - If a big 40ms or so occasional delay is seen in operations against HBase, - try the Nagles' setting. For example, see the user mailing list thread, - Inconsistent scan performance with caching set to 1 - and the issue cited therein where setting notcpdelay improved scan speeds. You might also - see the graphs on the tail of HBASE-7008 Set scanner caching to a better default - where our Lars Hofhansl tries various data sizes w/ Nagle's on and off measuring the effect. -

- Better Mean Time to Recover (MTTR) - This section is about configurations that will make servers come back faster after a fail. - See the Deveraj Das an Nicolas Liochon blog post - Introduction to HBase Mean Time to Recover (MTTR) - for a brief introduction. - The issue HBASE-8354 forces Namenode into loop with lease recovery requests - is messy but has a bunch of good discussion toward the end on low timeouts and how to effect faster recovery including citation of fixes - added to HDFS. Read the Varun Sharma comments. The below suggested configurations are Varun's suggestions distilled and tested. Make sure you are - running on a late-version HDFS so you have the fixes he refers too and himself adds to HDFS that help HBase MTTR - (e.g. HDFS-3703, HDFS-3712, and HDFS-4791 -- hadoop 2 for sure has them and late hadoop 1 has some). - Set the following in the RegionServer. - - - - hbase.lease.recovery.dfs.timeout - 23000 - How much time we allow elapse between calls to recover lease. - Should be larger than the dfs timeout. - - - dfs.client.socket-timeout - 10000 - Down the DFS timeout from 60 to 10 seconds. - -]]> - - And on the namenode/datanode side, set the following to enable 'staleness' introduced - in HDFS-3703, HDFS-3912. - - dfs.client.socket-timeout - 10000 - Down the DFS timeout from 60 to 10 seconds. - - - dfs.datanode.socket.write.timeout - 10000 - Down the DFS timeout from 8 * 60 to 10 seconds. - - - ipc.client.connect.timeout - 3000 - Down from 60 seconds to 3. - - - ipc.client.connect.max.retries.on.timeouts - 2 - Down from 45 seconds to 3 (2 == 3 retries). - - - dfs.namenode.avoid.read.stale.datanode - true - Enable stale state in hdfs - - - dfs.namenode.stale.datanode.interval - 20000 - Down from default 30 seconds - - - dfs.namenode.avoid.write.stale.datanode - true - Enable stale state in hdfs - -]]> -

- -

- JMX - JMX(Java Management Extensions) provides built-in instrumentation that enables you - to monitor and manage the Java VM. To enable monitoring and management from remote - systems, you need to set system property com.sun.management.jmxremote.port(the port - number through which you want to enable JMX RMI connections) when you start the Java VM. - See - official document for more information. Historically, besides above port mentioned, - JMX opens 2 additional random TCP listening ports, which could lead to port conflict - problem.(See HBASE-10289 - for details) - - As an alternative, You can use the coprocessor-based JMX implementation provided - by HBase. To enable it in 0.99 or above, add below property in - hbase-site.xml: - - hbase.coprocessor.regionserver.classes - org.apache.hadoop.hbase.JMXListener - -]]> - NOTE: DO NOT set com.sun.management.jmxremote.port for Java VM at the same time. - - Currently it supports Master and RegionServer Java VM. The reason why you only - configure coprocessor for 'regionserver' is that, starting from HBase 0.99, - a Master IS also a RegionServer. (See HBASE-10569 - for more information.) - By default, the JMX listens on TCP port 10102, you can further configure the port - using below properties: - - - regionserver.rmi.registry.port - 61130 - - - regionserver.rmi.connector.port - 61140 - -]]> - The registry port can be shared with connector port in most cases, so you only - need to configure regionserver.rmi.registry.port. However if you want to use SSL - communication, the 2 ports must be configured to different values. - - - By default the password authentication and SSL communication is disabled. - To enable password authentication, you need to update hbase-env.sh - like below: - -export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.authenticate=true \ - -Dcom.sun.management.jmxremote.password.file=your_password_file \ - -Dcom.sun.management.jmxremote.access.file=your_access_file" - -export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE " -export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE " - - See example password/access file under $JRE_HOME/lib/management. - - - To enable SSL communication with password authentication, follow below steps: - -#1. generate a key pair, stored in myKeyStore -keytool -genkey -alias jconsole -keystore myKeyStore - -#2. export it to file jconsole.cert -keytool -export -alias jconsole -keystore myKeyStore -file jconsole.cert - -#3. copy jconsole.cert to jconsole client machine, import it to jconsoleKeyStore -keytool -import -alias jconsole -keystore jconsoleKeyStore -file jconsole.cert - - And then update hbase-env.sh like below: - -export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=true \ - -Djavax.net.ssl.keyStore=/home/tianq/myKeyStore \ - -Djavax.net.ssl.keyStorePassword=your_password_in_step_1 \ - -Dcom.sun.management.jmxremote.authenticate=true \ - -Dcom.sun.management.jmxremote.password.file=your_password file \ - -Dcom.sun.management.jmxremote.access.file=your_access_file" - -export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE " -export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE " - - - Finally start jconsole on client using the key store: - -jconsole -J-Djavax.net.ssl.trustStore=/home/tianq/jconsoleKeyStore - - - NOTE: for HBase 0.98, To enable the HBase JMX implementation on Master, you also - need to add below property in hbase-site.xml: - - hbase.coprocessor.master.classes - org.apache.hadoop.hbase.JMXListener - -]]> - The corresponding properties for port configuration are master.rmi.registry.port - (by default 10101) and master.rmi.connector.port(by default the same as registry.port) - -

- -

- Dynamic Configuration - Changing Configuration Without Restarting Servers - Since HBase 1.0.0, it is possible to change a subset of the configuration without - requiring a server restart. In the hbase shell, there are new operators, - update_config and update_all_config that - will prompt a server or all servers to reload configuration. - Only a subset of all configurations can currently be changed in the running server. - Here is an incomplete list: - hbase.regionserver.thread.compaction.large, - hbase.regionserver.thread.compaction.small, - hbase.regionserver.thread.split, - hbase.regionserver.thread.merge, as well as compaction - policy and configurations and adjustment to offpeak hours. - For the full list consult the patch attached to - HBASE-12147 Porting Online Config Change from 89-fb. - - -