hbase-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From st...@apache.org
Subject svn commit: r1029952 - in /hbase/trunk: CHANGES.txt src/docbkx/book.xml
Date Tue, 02 Nov 2010 04:57:26 GMT
Author: stack
Date: Tue Nov  2 04:57:25 2010
New Revision: 1029952

URL: http://svn.apache.org/viewvc?rev=1029952&view=rev
HBASE-1932  Encourage use of 'lzo' compression... add the wiki page to getting started


Modified: hbase/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hbase/trunk/CHANGES.txt?rev=1029952&r1=1029951&r2=1029952&view=diff
--- hbase/trunk/CHANGES.txt (original)
+++ hbase/trunk/CHANGES.txt Tue Nov  2 04:57:25 2010
@@ -640,6 +640,8 @@ Release 0.21.0 - Unreleased
    HBASE-3179  Enable ReplicationLogsCleaner only if replication is,
                and fix its test
    HBASE-3185  User-triggered compactions are triggering splits!
+   HBASE-1932  Encourage use of 'lzo' compression... add the wiki page to
+               getting started

Modified: hbase/trunk/src/docbkx/book.xml
URL: http://svn.apache.org/viewvc/hbase/trunk/src/docbkx/book.xml?rev=1029952&r1=1029951&r2=1029952&view=diff
--- hbase/trunk/src/docbkx/book.xml (original)
+++ hbase/trunk/src/docbkx/book.xml Tue Nov  2 04:57:25 2010
@@ -230,6 +230,7 @@ stopping hbase...............</programli
       different HBase run modes: standalone, what is described above in <link
       linkend="quickstart">Quick Start,</link> pseudo-distributed where all
       daemons run on a single server, and distributed.</para>
+      <para>Be sure to read the <link linkend="important_configurations">Important
@@ -242,7 +243,6 @@ stopping hbase...............</programli
     <title><filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename></title>
     <para>What are these?
     Not all configuration options make it out to
     <filename>hbase-default.xml</filename>.  Configuration
@@ -250,38 +250,95 @@ stopping hbase...............</programli
     in code; the only way to turn up the configurations is
     via a reading of the source code.
     <!--The file hbase-default.xml is generated as part of
     the build of the hbase site.  See the hbase pom.xml.
     The generated file is a docbook section with a glossary
     in it-->
     <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
       href="../../target/site/hbase-default.xml" />
-      </section>
+    </section>
-      <section>
-      <title>Noteworthy Configuration</title>
-      <para>Below we review a couple of the key configurations.
-      We'll list those you must to change to suit your context
-      and others that you should review and consider moving on
-      from defaults after guaging your deploys load and query profiles.
+      <section xml:id="important_configurations">
+      <title>The Important Configurations</title>
+      <para>Below we list the important Configurations.  We've divided this section
+      required configuration and worth-a-look recommended configs.
-      <section>
+      <section xml:id="required_configuration"><title>Required Configurations</title>
+      <para>Here are some configurations you must configure to suit
+      your deploy.
+      </para>
+      <section xml:id="ulimit">
+      <title><varname>ulimit</varname></title>
+      <para>HBase is a database, it uses a lot of files at the same time.
+      The default ulimit -n of 1024 on *nix systems is insufficient.
+      Any significant amount of loading will lead you to 
+      <link xlink:href="http://wiki.apache.org/hadoop/Hbase/FAQ#A6">FAQ: Why do I see
"java.io.IOException...(Too many open files)" in my logs?</link>.
+      You will also notice errors like:
+      <programlisting>2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient:
Exception increateBlockOutputStream java.io.EOFException
+      2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901
+      </programlisting>
+      Do yourself a favor and change the upper bound on the number of file descriptors.
+      Set it to north of 10k.  See the above referenced FAQ for how.</para>
+      <para>To be clear, upping the file descriptors for the user who is
+      running the HBase process is an operating system configuration, not an
+      HBase configuration.
+      </para>
+      </section>
+      <section xml:id="dfs.datanode.max.xcievers">
+      <title><varname>dfs.datanode.max.xcievers</varname></title>
+      <para>
+      Hadoop HDFS has an upper bound of files that it will serve at one same time,
+      called <varname>xcievers</varname> (yes, this is misspelled). Again, before
+      doing any loading, make sure you have configured Hadoop's <filename>conf/hdfs-site.xml</filename>
+      setting the <varname>xceivers</varname> value to at least the following:
+      <programlisting>
+      &lt;property&gt;
+        &lt;name&gt;dfs.datanode.max.xcievers&lt;/name&gt;
+        &lt;value&gt;2047&lt;/value&gt;
+      &lt;/property&gt;
+      </programlisting>
+      </para>
+      </section>
+      </section>
+      <section xml:id="recommended_configurations"><title>Recommended Configuations</title>
+      <section xml:id="lzo">
       <title>LZO compression</title>
       <para>You should consider enabling LZO compression.  Its
       near-frictionless and in most all cases boosts performance.
-      To enable LZO, TODO...
+      <para>Unfortunately, HBase cannot ship with LZO because of
+      the licensing issues; HBase is Apache-licensed, LZO is GPL.
+      Therefore LZO install is to be done post-HBase install.
+      See the <link xlink:href="http://wiki.apache.org/hadoop/UsingLzoCompression">Using
LZO Compression</link>
+      wiki page for how to make LZO work with HBase.
+      </para>
+      <para>A common problem users run into when using LZO is that while initial
+      setup of the cluster runs smooth, a month goes by and some sysadmin goes to
+      add a machine to the cluster only they'll have forgotten to do the LZO
+      fixup on the new machine.  In versions since HBase 0.90.0, we should
+      fail in a way that makes it plain what the problem is, but maybe not.
+      Remember you read this paragraph<footnote><para>See
+      <link linkend="hbase.regionserver.codec">hbase.regionserver.codec</link>
+      for a feature to help protect against failed LZO install</para></footnote>.
+      </para>
+      </section>
@@ -1201,7 +1258,7 @@ stopping hbase...............</programli
     xlink:href="http://en.wikipedia.org/wiki/Write-ahead_logging"> Write-Ahead
-    <para>Each RegionServer adds updates to its <link linkend="???">WAL</link>
+    <para>Each RegionServer adds updates to its Write-ahead Log (WAL)
     first, and then to memory.</para>

View raw message