Return-Path: X-Original-To: apmail-hbase-commits-archive@www.apache.org Delivered-To: apmail-hbase-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 06FE911081 for ; Fri, 1 Aug 2014 16:40:16 +0000 (UTC) Received: (qmail 35980 invoked by uid 500); 1 Aug 2014 16:40:15 -0000 Delivered-To: apmail-hbase-commits-archive@hbase.apache.org Received: (qmail 35854 invoked by uid 500); 1 Aug 2014 16:40:15 -0000 Mailing-List: contact commits-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list commits@hbase.apache.org Received: (qmail 35742 invoked by uid 99); 1 Aug 2014 16:40:15 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Aug 2014 16:40:15 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 619DB9BD8E7; Fri, 1 Aug 2014 16:40:15 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: stack@apache.org To: commits@hbase.apache.org Date: Fri, 01 Aug 2014 16:40:17 -0000 Message-Id: <3de22229f3f149d3842d011a6d99f20b@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [3/3] git commit: HBASE-11640 Add syntax highlighting support to HBase Ref Guide programlistings (Misty Stanley-Jones) HBASE-11640 Add syntax highlighting support to HBase Ref Guide programlistings (Misty Stanley-Jones) Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/24b5fa7f Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/24b5fa7f Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/24b5fa7f Branch: refs/heads/master Commit: 24b5fa7f0c442b54157d961bf47d912455275151 Parents: 78d47cc Author: stack Authored: Fri Aug 1 09:39:56 2014 -0700 Committer: stack Committed: Fri Aug 1 09:40:03 2014 -0700 ---------------------------------------------------------------------- .../appendix_contributing_to_documentation.xml | 41 ++++-- src/main/docbkx/book.xml | 146 +++++++++---------- src/main/docbkx/case_studies.xml | 4 +- src/main/docbkx/configuration.xml | 44 +++--- src/main/docbkx/cp.xml | 2 +- src/main/docbkx/customization.xsl | 1 + src/main/docbkx/developer.xml | 130 ++++++++--------- src/main/docbkx/getting_started.xml | 36 ++--- src/main/docbkx/hbase_apis.xml | 4 +- src/main/docbkx/ops_mgt.xml | 82 +++++------ src/main/docbkx/performance.xml | 16 +- src/main/docbkx/preface.xml | 2 +- src/main/docbkx/schema_design.xml | 12 +- src/main/docbkx/security.xml | 78 +++++----- src/main/docbkx/thrift_filter_language.xml | 86 +++++------ src/main/docbkx/tracing.xml | 12 +- src/main/docbkx/troubleshooting.xml | 26 ++-- src/main/docbkx/upgrading.xml | 8 +- src/main/docbkx/zookeeper.xml | 28 ++-- 19 files changed, 391 insertions(+), 367 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hbase/blob/24b5fa7f/src/main/docbkx/appendix_contributing_to_documentation.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/appendix_contributing_to_documentation.xml b/src/main/docbkx/appendix_contributing_to_documentation.xml index 080525a..2f19c7b 100644 --- a/src/main/docbkx/appendix_contributing_to_documentation.xml +++ b/src/main/docbkx/appendix_contributing_to_documentation.xml @@ -107,7 +107,7 @@ For each issue you work on, create a new branch. One convention that works well for naming the branches is to name a given branch the same as the JIRA it relates to: - $ git checkout -b HBASE-123456 + $ git checkout -b HBASE-123456 Make your suggested changes on your branch, committing your changes to your @@ -123,8 +123,8 @@ sure you have built HBase at least once, in order to fetch all the Maven dependencies you need. - $ mvn clean install -DskipTests # Builds HBase - $ mvn clean site -DskipTests # Builds the website and documentation + $ mvn clean install -DskipTests # Builds HBase + $ mvn clean site -DskipTests # Builds the website and documentation If any errors occur, address them. @@ -132,7 +132,7 @@ the area of the code you are working in has had a lot of changes lately, make sure you rebase your branch against the remote master and take care of any conflicts before submitting your patch. - + $ git checkout HBASE-123456 $ git rebase origin/master @@ -141,7 +141,7 @@ $ git rebase origin/master Generate your patch against the remote master. Run the following command from the top level of your git repository (usually called hbase): - $ git diff --no-prefix origin/master > HBASE-123456.patch + $ git diff --no-prefix origin/master > HBASE-123456.patch The name of the patch should contain the JIRA ID. Look over the patch file to be sure that you did not change any additional files by accident and that there are no other surprises. When you are satisfied, attach the patch to the JIRA and @@ -227,7 +227,7 @@ $ git rebase origin/master recommended that you use a <figure> Docbook element for an image. This allows screen readers to navigate to the image and also provides alternative text for the image. The following is an example of a <figure> element. - + HFile Version 1 @@ -295,7 +295,7 @@ $ git rebase origin/master render as block-level elements (they take the whole width of the page), it is better to mark them up as siblings to the paragraphs around them, like this: - This is the paragraph. + This is the paragraph. This is an admonition which occurs after the paragraph. ]]> @@ -312,7 +312,7 @@ $ git rebase origin/master consist of things other than plain text, they need to be wrapped in some element. If they are plain text, they need to be inclosed in <para> tags. This is tedious but necessary for validity. - + This is a paragraph. @@ -367,7 +367,7 @@ $ git rebase origin/master the content. Also, to avoid having an extra blank line at the beginning of the programlisting output, do not put the CDATA element on its own line. For example: - + case $1 in --cleanZk|--cleanHdfs|--cleanAll) matches="yes" ;; @@ -396,6 +396,29 @@ esac especially if you use GUI mode in the editor. + + + Syntax Highlighting + + + The HBase Reference Guide uses the XSLT Syntax Highlighting Maven module for syntax highlighting. + To enable syntax highlighting for a given <programlisting> or + <screen> (or possibly other elements), add the attribute + language=LANGUAGE_OF_CHOICE + to the element, as in the following example: + + bar + foo +]]> + Several syntax types are supported. The most interesting ones for the + HBase Reference Guide are java, xml, + sql, and bourne (for BASH shell + output or Linux command-line examples). + + http://git-wip-us.apache.org/repos/asf/hbase/blob/24b5fa7f/src/main/docbkx/book.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/book.xml b/src/main/docbkx/book.xml index 0564354..36f2257 100644 --- a/src/main/docbkx/book.xml +++ b/src/main/docbkx/book.xml @@ -300,25 +300,25 @@ A namespace can be created, removed or altered. Namespace membership is determined during table creation by specifying a fully-qualified table name of the form: - :]]> + :
]]> Examples - + #Create a namespace create_namespace 'my_ns' - + #create my_table in my_ns namespace create 'my_ns:my_table', 'fam' - + #drop namespace drop_namespace 'my_ns' - + #alter namespace alter_namespace 'my_ns', {METHOD => 'set', 'PROPERTY_NAME' => 'PROPERTY_VALUE'} @@ -340,7 +340,7 @@ alter_namespace 'my_ns', {METHOD => 'set', 'PROPERTY_NAME' => 'PROPERTY_VALUE'} Examples - + #namespace=foo and table qualifier=bar create 'foo:bar', 'fam' @@ -429,7 +429,7 @@ create 'bar', 'fam' populated with rows with keys "row1", "row2", "row3", and then another set of rows with the keys "abc1", "abc2", and "abc3". The following example shows how startRow and stopRow can be applied to a Scan instance to return the rows beginning with "row". - + public static final byte[] CF = "cf".getBytes(); public static final byte[] ATTR = "attr".getBytes(); ... @@ -562,7 +562,7 @@ try { xml:id="default_get_example"> Default Get Example The following Get will only retrieve the current version of the row - + public static final byte[] CF = "cf".getBytes(); public static final byte[] ATTR = "attr".getBytes(); ... @@ -575,7 +575,7 @@ byte[] b = r.getValue(CF, ATTR); // returns current version of value xml:id="versioned_get_example"> Versioned Get Example The following Get will return the last 3 versions of the row. - + public static final byte[] CF = "cf".getBytes(); public static final byte[] ATTR = "attr".getBytes(); ... @@ -603,7 +603,7 @@ List<KeyValue> kv = r.getColumn(CF, ATTR); // returns all versions of thi Implicit Version Example The following Put will be implicitly versioned by HBase with the current time. - + public static final byte[] CF = "cf".getBytes(); public static final byte[] ATTR = "attr".getBytes(); ... @@ -616,7 +616,7 @@ htable.put(put); xml:id="explicit_version_example"> Explicit Version Example The following Put has the version timestamp explicitly set. - + public static final byte[] CF = "cf".getBytes(); public static final byte[] ATTR = "attr".getBytes(); ... @@ -815,7 +815,7 @@ htable.put(put); Be sure to use the correct version of the HBase JAR for your system. The backticks (` symbols) cause ths shell to execute the sub-commands, setting the CLASSPATH as part of the command. This example assumes you use a BASH-compatible shell. - $ HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-0.90.0.jar rowcounter usertable + $ HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-0.90.0.jar rowcounter usertable When the command runs, internally, the HBase JAR finds the dependencies it needs for zookeeper, guava, and its other dependencies on the passed HADOOP_CLASSPATH and adds the JARs to the MapReduce job configuration. See the source at @@ -826,7 +826,7 @@ htable.put(put); java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.RowCounter$RowCounterMapper If this occurs, try modifying the command as follows, so that it uses the HBase JARs from the target/ directory within the build environment. - $ HADOOP_CLASSPATH=${HBASE_HOME}/target/hbase-0.90.0-SNAPSHOT.jar:`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/target/hbase-0.90.0-SNAPSHOT.jar rowcounter usertable + $ HADOOP_CLASSPATH=${HBASE_HOME}/target/hbase-0.90.0-SNAPSHOT.jar:`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/target/hbase-0.90.0-SNAPSHOT.jar rowcounter usertable Notice to Mapreduce users of HBase 0.96.1 and above @@ -876,14 +876,14 @@ Exception in thread "main" java.lang.IllegalAccessError: class HADOOP_CLASSPATH environment variable at job submission time. When launching jobs that package their dependencies, all three of the following job launching commands satisfy this requirement: - + $ HADOOP_CLASSPATH=/path/to/hbase-protocol.jar:/path/to/hbase/conf hadoop jar MyJob.jar MyJobMainClass $ HADOOP_CLASSPATH=$(hbase mapredcp):/path/to/hbase/conf hadoop jar MyJob.jar MyJobMainClass $ HADOOP_CLASSPATH=$(hbase classpath) hadoop jar MyJob.jar MyJobMainClass For jars that do not package their dependencies, the following command structure is necessary: - + $ HADOOP_CLASSPATH=$(hbase mapredcp):/etc/hbase/conf hadoop jar MyApp.jar MyJobMainClass -libjars $(hbase mapredcp | tr ':' ',') ... See also HADOOP_CLASSPATH=$(hbase mapredcp):/etc/hbase/conf hadoop jar MyApp The HBase JAR also serves as a Driver for some bundled mapreduce jobs. To learn about the bundled MapReduce jobs, run the following command. - $ ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-0.90.0-SNAPSHOT.jar + $ ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-0.90.0-SNAPSHOT.jar An example program must be given as the first argument. Valid program names are: copytable: Export a table from local cluster to peer cluster @@ -910,7 +910,7 @@ Valid program names are: Each of the valid program names are bundled MapReduce jobs. To run one of the jobs, model your command after the following example. - $ ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-0.90.0-SNAPSHOT.jar rowcounter myTable + $ ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-0.90.0-SNAPSHOT.jar rowcounter myTable
@@ -972,7 +972,7 @@ Valid program names are: xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/RowCounter.html">RowCounter MapReduce job uses TableInputFormat and does a count of all rows in the specified table. To run it, use the following command: - $ ./bin/hadoop jar hbase-X.X.X.jar + $ ./bin/hadoop jar hbase-X.X.X.jar This will invoke the HBase MapReduce Driver class. Select rowcounter from the choice of jobs offered. This will print rowcouner usage advice to standard output. Specify the tablename, @@ -1011,7 +1011,7 @@ Valid program names are: The following is an example of using HBase as a MapReduce source in read-only manner. Specifically, there is a Mapper instance but no Reducer, and nothing is being emitted from the Mapper. There job would be defined as follows... - + Configuration config = HBaseConfiguration.create(); Job job = new Job(config, "ExampleRead"); job.setJarByClass(MyReadJob.class); // class that contains mapper @@ -1038,7 +1038,7 @@ if (!b) { ...and the mapper instance would extend TableMapper... - + public static class MyMapper extends TableMapper<Text, Text> { public void map(ImmutableBytesWritable row, Result value, Context context) throws InterruptedException, IOException { @@ -1052,7 +1052,7 @@ public static class MyMapper extends TableMapper<Text, Text> { HBase MapReduce Read/Write Example The following is an example of using HBase both as a source and as a sink with MapReduce. This example will simply copy data from one table to another. - + Configuration config = HBaseConfiguration.create(); Job job = new Job(config,"ExampleReadWrite"); job.setJarByClass(MyReadWriteJob.class); // class that contains mapper @@ -1091,7 +1091,7 @@ if (!b) { The following is the example mapper, which will create a Put and matching the input Result and emit it. Note: this is what the CopyTable utility does. - + public static class MyMapper extends TableMapper<ImmutableBytesWritable, Put> { public void map(ImmutableBytesWritable row, Result value, Context context) throws IOException, InterruptedException { @@ -1125,7 +1125,7 @@ public static class MyMapper extends TableMapper<ImmutableBytesWritable, Put& The following example uses HBase as a MapReduce source and sink with a summarization step. This example will count the number of distinct instances of a value in a table and write those summarized counts in another table. - + Configuration config = HBaseConfiguration.create(); Job job = new Job(config,"ExampleSummary"); job.setJarByClass(MySummaryJob.class); // class that contains mapper and reducer @@ -1156,7 +1156,7 @@ if (!b) { In this example mapper a column with a String-value is chosen as the value to summarize upon. This value is used as the key to emit from the mapper, and an IntWritable represents an instance counter. - + public static class MyMapper extends TableMapper<Text, IntWritable> { public static final byte[] CF = "cf".getBytes(); public static final byte[] ATTR1 = "attr1".getBytes(); @@ -1174,7 +1174,7 @@ public static class MyMapper extends TableMapper<Text, IntWritable> { In the reducer, the "ones" are counted (just like any other MR example that does this), and then emits a Put. - + public static class MyTableReducer extends TableReducer<Text, IntWritable, ImmutableBytesWritable> { public static final byte[] CF = "cf".getBytes(); public static final byte[] COUNT = "count".getBytes(); @@ -1199,7 +1199,7 @@ public static class MyTableReducer extends TableReducer<Text, IntWritable, Im This very similar to the summary example above, with exception that this is using HBase as a MapReduce source but HDFS as the sink. The differences are in the job setup and in the reducer. The mapper remains the same. - + Configuration config = HBaseConfiguration.create(); Job job = new Job(config,"ExampleSummaryToFile"); job.setJarByClass(MySummaryFileJob.class); // class that contains mapper and reducer @@ -1228,7 +1228,7 @@ if (!b) { As stated above, the previous Mapper can run unchanged with this example. As for the Reducer, it is a "generic" Reducer instead of extending TableMapper and emitting Puts. - + public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { @@ -1268,7 +1268,7 @@ if (!b) { reducers. Neither is right or wrong, it depends on your use-case. Recognize that the more reducers that are assigned to the job, the more simultaneous connections to the RDBMS will be created - this will scale, but only to a point. - + public static class MyRdbmsReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private Connection c = null; @@ -1299,7 +1299,7 @@ if (!b) { Although the framework currently allows one HBase table as input to a MapReduce job, other HBase tables can be accessed as lookup tables, etc., in a MapReduce job via creating an HTable instance in the setup method of the Mapper. - public class MyMapper extends TableMapper<Text, LongWritable> { + public class MyMapper extends TableMapper<Text, LongWritable> { private HTable myOtherTable; public void setup(Context context) { @@ -1519,11 +1519,11 @@ if (!b) { xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HBaseConfiguration">HBaseConfiguration instance. This will ensure sharing of ZooKeeper and socket instances to the RegionServers which is usually what you want. For example, this is preferred: - HBaseConfiguration conf = HBaseConfiguration.create(); + HBaseConfiguration conf = HBaseConfiguration.create(); HTable table1 = new HTable(conf, "myTable"); HTable table2 = new HTable(conf, "myTable"); as opposed to this: - HBaseConfiguration conf1 = HBaseConfiguration.create(); + HBaseConfiguration conf1 = HBaseConfiguration.create(); HTable table1 = new HTable(conf1, "myTable"); HBaseConfiguration conf2 = HBaseConfiguration.create(); HTable table2 = new HTable(conf2, "myTable"); @@ -1537,7 +1537,7 @@ HTable table2 = new HTable(conf2, "myTable"); the following example: Pre-Creating a <code>HConnection</code> - // Create a connection to the cluster. + // Create a connection to the cluster. HConnection connection = HConnectionManager.createConnection(Configuration); HTableInterface table = connection.getTable("myTable"); // use table as needed, the table returned is lightweight @@ -1594,7 +1594,7 @@ connection.close(); represents a list of Filters with a relationship of FilterList.Operator.MUST_PASS_ALL or FilterList.Operator.MUST_PASS_ONE between the Filters. The following example shows an 'or' between two Filters (checking for either 'my value' or 'my other value' on the same attribute). - + FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ONE); SingleColumnValueFilter filter1 = new SingleColumnValueFilter( cf, @@ -1627,7 +1627,7 @@ scan.setFilter(list); ), inequality (CompareOp.NOT_EQUAL), or ranges (e.g., CompareOp.GREATER). The following is example of testing equivalence a column to a String value "my value"... - + SingleColumnValueFilter filter = new SingleColumnValueFilter( cf, column, @@ -1650,7 +1650,7 @@ scan.setFilter(filter); RegexStringComparator supports regular expressions for value comparisons. - + RegexStringComparator comp = new RegexStringComparator("my."); // any value that starts with 'my' SingleColumnValueFilter filter = new SingleColumnValueFilter( cf, @@ -1671,7 +1671,7 @@ scan.setFilter(filter); xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SubstringComparator.html">SubstringComparator can be used to determine if a given substring exists in a value. The comparison is case-insensitive. - + SubstringComparator comp = new SubstringComparator("y val"); // looking for 'my value' SingleColumnValueFilter filter = new SingleColumnValueFilter( cf, @@ -1728,7 +1728,7 @@ scan.setFilter(filter); Note: The same column qualifier can be used in different column families. This filter returns all matching columns. Example: Find all columns in a row and family that start with "abc" - + HTableInterface t = ...; byte[] row = ...; byte[] family = ...; @@ -1758,7 +1758,7 @@ rs.close(); prefixes. It can be used to efficiently get discontinuous sets of columns from very wide rows. Example: Find all columns in a row and family that start with "abc" or "xyz" - + HTableInterface t = ...; byte[] row = ...; byte[] family = ...; @@ -1791,7 +1791,7 @@ rs.close(); filter returns all matching columns. Example: Find all columns in a row and family between "bbbb" (inclusive) and "bbdd" (inclusive) - + HTableInterface t = ...; byte[] row = ...; byte[] family = ...; @@ -2018,7 +2018,7 @@ rs.close(); was accessed. Catalog tables are configured like this. This group is the last one considered during evictions. To mark a column family as in-memory, call - HColumnDescriptor.setInMemory(true); if creating a table from java, + HColumnDescriptor.setInMemory(true); if creating a table from java, or set IN_MEMORY => true when creating or altering a table in the shell: e.g. hbase(main):003:0> create 't', {NAME => 'f', IN_MEMORY => 'true'} @@ -2218,7 +2218,7 @@ rs.close(); Next, add the following configuration to the RegionServer's hbase-site.xml. - + hbase.bucketcache.ioengine offheap @@ -2461,7 +2461,7 @@ rs.close(); ZooKeeper splitlog node (/hbase/splitlog) as tasks. You can view the contents of the splitlog by issuing the following zkcli command. Example output is shown. - ls /hbase/splitlog + ls /hbase/splitlog [hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost8.sample.com%2C57020%2C1340474893275-splitting%2Fhost8.sample.com%253A57020.1340474893900, hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost3.sample.com%2C57020%2C1340474893299-splitting%2Fhost3.sample.com%253A57020.1340474893931, hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost4.sample.com%2C57020%2C1340474893287-splitting%2Fhost4.sample.com%253A57020.1340474893946] @@ -2846,7 +2846,7 @@ ctime = Sat Jun 23 11:13:40 PDT 2012 Typically a custom split policy should extend HBase's default split policy: ConstantSizeRegionSplitPolicy. The policy can set globally through the HBaseConfiguration used or on a per table basis: - + HTableDescriptor myHtd = ...; myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName()); @@ -2867,7 +2867,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( opens merged region on the regionserver and reports the merge to Master at last. An example of region merges in the hbase shell - $ hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME' + $ hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME' hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', true It's an asynchronous operation and call returns immediately without waiting merge completed. @@ -2969,10 +2969,10 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( To view a textualized version of hfile content, you can do use the org.apache.hadoop.hbase.io.hfile.HFile - tool. Type the following to see usage:$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.io.hfile.HFile For + tool. Type the following to see usage:$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.io.hfile.HFile For example, to view the content of the file hdfs://10.81.47.41:8020/hbase/TEST/1418428042/DSMP/4759508618286845475, - type the following: $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f hdfs://10.81.47.41:8020/hbase/TEST/1418428042/DSMP/4759508618286845475 If + type the following: $ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f hdfs://10.81.47.41:8020/hbase/TEST/1418428042/DSMP/4759508618286845475 If you leave off the option -v to see just a summary on the hfile. See usage for other things to do with the HFile tool. @@ -3818,7 +3818,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( Run one of following commands in the HBase shell. Replace the table name orders_table with the name of your table. - + alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}} create 'orders_table', 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} @@ -3842,7 +3842,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( Set the hbase.hstore.engine.class option to either nil or org.apache.hadoop.hbase.regionserver.DefaultStoreEngine. Either option has the same effect. - + alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => ''} @@ -3861,7 +3861,7 @@ myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName( column family, after disabling the table. If you use HBase shell, the general command pattern is as follows: - + alter 'orders_table', CONFIGURATION => {'key' => 'value', ..., 'key' => 'value'}}
{'key' => 'value', ..., 'key' => 'value'} where importtsv or your MapReduce job put its results, and the table name to import into. For example: - $ hadoop jar hbase-VERSION.jar completebulkload [-c /path/to/hbase/config/hbase-site.xml] /user/todd/myoutput mytable + $ hadoop jar hbase-VERSION.jar completebulkload [-c /path/to/hbase/config/hbase-site.xml] /user/todd/myoutput mytable The -c config-file option can be used to specify a file containing the appropriate hbase parameters (e.g., hbase-site.xml) if @@ -4143,7 +4143,7 @@ alter 'orders_table', CONFIGURATION => {'key' => 'value', ..., 'key' => 'value'} Timeline Consistency With this feature, HBase introduces a Consistency definition, which can be provided per read operation (get or scan). - + public enum Consistency { STRONG, TIMELINE @@ -4254,7 +4254,7 @@ public enum Consistency {
Server side properties - hbase.regionserver.storefile.refresh.period 0 @@ -4274,7 +4274,7 @@ public enum Consistency { Client side properties Ensure to set the following for all clients (and servers) that will use region replicas. - hbase.ipc.client.allowsInterrupt true @@ -4325,7 +4325,7 @@ flush 't1'
Java - get 't1','r6', {CONSISTENCY => "TIMELINE"} ]]> You can simulate a region server pausing or becoming unavailable and do a read from the secondary replica: - hbase(main):001:0> get 't1','r6', {CONSISTENCY => "TIMELINE"} @@ -4376,14 +4376,14 @@ hbase> scan 't1', {CONSISTENCY => 'TIMELINE'} Java You can set set the consistency for Gets and Scans and do requests as follows. - You can also pass multiple gets: - And Scans: - You can inspect whether the results are coming from primary region or not by calling the Result.isStale() method: - Running hbck to identify inconsistencies To check to see if your HBase cluster has corruptions, run hbck against your HBase cluster: - + $ ./bin/hbase hbck @@ -4661,13 +4661,13 @@ A run of hbck will report a list of inconsistencies along with a brief descripti tables affected. The using the -details option will report more details including a representative listing of all the splits present in all the tables. - + $ ./bin/hbase hbck -details If you just want to know if some tables are corrupted, you can limit hbck to identify inconsistencies in only specific tables. For example the following command would only attempt to check table TableFoo and TableBar. The benefit is that hbck will run in less time. - + $ ./bin/hbase hbck TableFoo TableBar
@@ -4726,12 +4726,12 @@ assigned or multiply assigned regions. To fix deployment and assignment problems you can run this command: - + $ ./bin/hbase hbck -fixAssignments To fix deployment and assignment problems as well as repairing incorrect meta rows you can run this command: - + $ ./bin/hbase hbck -fixAssignments -fixMeta There are a few classes of table integrity problems that are low risk repairs. The first two are @@ -4743,12 +4743,12 @@ The third low-risk class is hdfs region holes. This can be repaired by using the If holes are detected you can use -fixHdfsHoles and should include -fixMeta and -fixAssignments to make the new region consistent. - + $ ./bin/hbase hbck -fixAssignments -fixMeta -fixHdfsHoles Since this is a common operation, we’ve added a the -repairHoles flag that is equivalent to the previous command: - + $ ./bin/hbase hbck -repairHoles If inconsistencies still remain after these steps, you most likely have table integrity problems @@ -4800,14 +4800,14 @@ integrity options. Finally, there are safeguards to limit repairs to only specific tables. For example the following command would only attempt to check and repair table TableFoo and TableBar. - + $ ./bin/hbase hbck -repair TableFoo TableBar
Special cases: Meta is not properly assigned There are a few special cases that hbck can handle as well. Sometimes the meta table’s only region is inconsistently assigned or deployed. In this case there is a special -fixMetaOnly option that can try to fix meta assignments. - + $ ./bin/hbase hbck -fixMetaOnly -fixAssignments
@@ -4825,7 +4825,7 @@ directory, loads as much information from region metadata files (.regioninfo fil from the file system. If the region metadata has proper table integrity, it sidelines the original root and meta table directories, and builds new ones with pointers to the region directories and their data. - + $ ./bin/hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair NOTE: This tool is not as clever as uberhbck but can be used to bootstrap repairs that uberhbck @@ -5085,7 +5085,7 @@ This option should not normally be used, and it is not in -fixAll. linkend="hbase.native.platform" />), you can make a symbolic link from HBase to the native Hadoop libraries. This assumes the two software installs are colocated. For example, if my 'platform' is Linux-amd64-64: - $ cd $HBASE_HOME + $ cd $HBASE_HOME $ mkdir lib/native $ ln -s $HADOOP_HOME/lib/native lib/native/Linux-amd64-64 Use the compression tool to check that LZ4 is installed on all nodes. Start up (or restart) @@ -5128,7 +5128,7 @@ hbase(main):003:0> alter 'TestTable', {NAME => 'info', COMPRESSION => CompressionTest You can use the CompressionTest tool to verify that your compressor is available to HBase: - + $ hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://host/path/to/hbase snappy
@@ -5192,7 +5192,7 @@ DESCRIPTION ENABLED parameter, usage advice is printed for each option. <command>LoadTestTool</command> Usage - Options: @@ -5248,7 +5248,7 @@ Options: Example Usage of LoadTestTool - + $ hbase org.apache.hadoop.hbase.util.LoadTestTool -write 1:10:100 -num_keys 1000000 -read 100:30 -num_tables 1 -data_block_encoding NONE -tn load_test_tool_NONE http://git-wip-us.apache.org/repos/asf/hbase/blob/24b5fa7f/src/main/docbkx/case_studies.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/case_studies.xml b/src/main/docbkx/case_studies.xml index 7824c7d..332caf8 100644 --- a/src/main/docbkx/case_studies.xml +++ b/src/main/docbkx/case_studies.xml @@ -145,7 +145,7 @@ some unusual anomalies, namely interface errors, overruns, framing errors. While not unheard of, these kinds of errors are exceedingly rare on modern hardware which is operating as it should: - + $ /sbin/ifconfig bond0 bond0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 inet addr:10.x.x.x Bcast:10.x.x.255 Mask:255.255.255.0 @@ -160,7 +160,7 @@ RX bytes:2416328868676 (2.4 TB) TX bytes:3464991094001 (3.4 TB) running an ICMP ping from an external host and observing round-trip-time in excess of 700ms, and by running ethtool(8) on the members of the bond interface and discovering that the active interface was operating at 100Mbs/, full duplex. - + $ sudo ethtool eth0 Settings for eth0: Supported ports: [ TP ] http://git-wip-us.apache.org/repos/asf/hbase/blob/24b5fa7f/src/main/docbkx/configuration.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/configuration.xml b/src/main/docbkx/configuration.xml index b0b2864..5949b0a 100644 --- a/src/main/docbkx/configuration.xml +++ b/src/main/docbkx/configuration.xml @@ -520,16 +520,16 @@ Index: pom.xml Type the following commands: - + - + Building against the hadoop 2 profile by running something like the following command: - $ mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests + $ mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests S S @@ -615,7 +615,7 @@ Index: pom.xml hbase-site.xml -- and on the serverside in hdfs-site.xml (The sync facility HBase needs is a subset of the append code path). - dfs.support.append true @@ -644,7 +644,7 @@ Index: pom.xml Hadoop's conf/hdfs-site.xml, setting the dfs.datanode.max.transfer.threads value to at least the following: - dfs.datanode.max.transfer.threads 4096 @@ -779,7 +779,7 @@ Index: pom.xml configuration parameters. Most HBase configuration directives have default values, which are used unless the value is overridden in the hbase-site.xml. See for more information. - hbase.rootdir @@ -891,7 +891,7 @@ node-c.example.com finally disable and drop your tables. To stop HBase after exiting the HBase shell enter - $ ./bin/stop-hbase.sh + $ ./bin/stop-hbase.sh stopping hbase............... Shutdown can take a moment to complete. It can take longer if your cluster is comprised of many machines. If you are running a distributed operation, be sure to wait until HBase @@ -1063,7 +1063,7 @@ slf4j-log4j (slf4j-log4j12-1.5.8.jar) zookeeper (zookeeper-3.4.2.jar) An example basic hbase-site.xml for client only might look as - follows: @@ -1090,7 +1090,7 @@ zookeeper (zookeeper-3.4.2.jar) hbase.X.X.X.jar). It is also possible to specify configuration directly without having to read from a hbase-site.xml. For example, to set the ZooKeeper ensemble for the cluster programmatically do as follows: - Configuration config = HBaseConfiguration.create(); + Configuration config = HBaseConfiguration.create(); config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zookeeper locally If multiple ZooKeeper instances make up your ZooKeeper ensemble, they may be specified in a comma-separated list (just as in the hbase-site.xml file). This @@ -1126,7 +1126,7 @@ config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zooke xml:id="hbase_site"> <filename>hbase-site.xml</filename> - + @@ -1140,7 +1140,7 @@ config.set("hbase.zookeeper.quorum", "localhost"); // Here we are running zooke hbase.zookeeper.property.dataDir /export/zookeeper - Property from ZooKeeper's config zoo.cfg. + Property from ZooKeeper config zoo.cfg. The directory where the snapshot is stored. @@ -1191,7 +1191,7 @@ example9 hbase-env.sh file. Here we are setting the HBase heap to be 4G instead of the default 1G. - + $ git diff hbase-env.sh diff --git a/conf/hbase-env.sh b/conf/hbase-env.sh @@ -1476,7 +1476,7 @@ index e70ebc6..96f8c27 100644 running on a late-version HDFS so you have the fixes he refers too and himself adds to HDFS that help HBase MTTR (e.g. HDFS-3703, HDFS-3712, and HDFS-4791 -- hadoop 2 for sure has them and late hadoop 1 has some). Set the following in the RegionServer. - + hbase.lease.recovery.dfs.timeout @@ -1493,7 +1493,7 @@ index e70ebc6..96f8c27 100644 And on the namenode/datanode side, set the following to enable 'staleness' introduced in HDFS-3703, HDFS-3912. - dfs.client.socket-timeout 10000 @@ -1550,7 +1550,7 @@ index e70ebc6..96f8c27 100644 As an alternative, You can use the coprocessor-based JMX implementation provided by HBase. To enable it in 0.99 or above, add below property in hbase-site.xml: - hbase.coprocessor.regionserver.classes org.apache.hadoop.hbase.JMXListener @@ -1566,7 +1566,7 @@ index e70ebc6..96f8c27 100644 By default, the JMX listens on TCP port 10102, you can further configure the port using below properties: - regionserver.rmi.registry.port 61130 @@ -1584,7 +1584,7 @@ index e70ebc6..96f8c27 100644 By default the password authentication and SSL communication is disabled. To enable password authentication, you need to update hbase-env.sh like below: - + export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.authenticate=true \ -Dcom.sun.management.jmxremote.password.file=your_password_file \ -Dcom.sun.management.jmxremote.access.file=your_access_file" @@ -1596,7 +1596,7 @@ export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE " To enable SSL communication with password authentication, follow below steps: - + #1. generate a key pair, stored in myKeyStore keytool -genkey -alias jconsole -keystore myKeyStore @@ -1607,10 +1607,10 @@ keytool -export -alias jconsole -keystore myKeyStore -file jconsole.cert keytool -import -alias jconsole -keystore jconsoleKeyStore -file jconsole.cert And then update hbase-env.sh like below: - + export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=true \ -Djavax.net.ssl.keyStore=/home/tianq/myKeyStore \ - -Djavax.net.ssl.keyStorePassword=your_password_in_step_#1 \ + -Djavax.net.ssl.keyStorePassword=your_password_in_step_1 \ -Dcom.sun.management.jmxremote.authenticate=true \ -Dcom.sun.management.jmxremote.password.file=your_password file \ -Dcom.sun.management.jmxremote.access.file=your_access_file" @@ -1620,13 +1620,13 @@ export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE " Finally start jconsole on client using the key store: - + jconsole -J-Djavax.net.ssl.trustStore=/home/tianq/jconsoleKeyStore NOTE: for HBase 0.98, To enable the HBase JMX implementation on Master, you also need to add below property in hbase-site.xml: - hbase.coprocessor.master.classes org.apache.hadoop.hbase.JMXListener http://git-wip-us.apache.org/repos/asf/hbase/blob/24b5fa7f/src/main/docbkx/cp.xml ---------------------------------------------------------------------- diff --git a/src/main/docbkx/cp.xml b/src/main/docbkx/cp.xml index 9cc0859..7062d7d 100644 --- a/src/main/docbkx/cp.xml +++ b/src/main/docbkx/cp.xml @@ -265,7 +265,7 @@ Example RegionObserver Configuration In this example, one RegionObserver is configured for all the HBase tables. - hbase.coprocessor.region.classes org.apache.hadoop.hbase.coprocessor.AggregateImplementation http://git-wip-us.apache.org/repos/asf/hbase/blob/24b5fa7f/src/main/docbkx/customization.xsl ---------------------------------------------------------------------- diff --git a/src/main/docbkx/customization.xsl b/src/main/docbkx/customization.xsl index 43d8df7..5d0ec2c 100644 --- a/src/main/docbkx/customization.xsl +++ b/src/main/docbkx/customization.xsl @@ -22,6 +22,7 @@ */ --> +