hbase-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From st...@apache.org
Subject svn commit: r1099696 - in /hbase/branches/0.90/src/docbkx: book.xml build.xml configuration.xml getting_started.xml performance.xml preface.xml shell.xml upgrading.xml
Date Thu, 05 May 2011 06:21:55 GMT
Author: stack
Date: Thu May  5 06:21:55 2011
New Revision: 1099696

URL: http://svn.apache.org/viewvc?rev=1099696&view=rev
Log:
Bring book up to TRUNK out on the branch

Modified:
    hbase/branches/0.90/src/docbkx/book.xml
    hbase/branches/0.90/src/docbkx/build.xml
    hbase/branches/0.90/src/docbkx/configuration.xml
    hbase/branches/0.90/src/docbkx/getting_started.xml
    hbase/branches/0.90/src/docbkx/performance.xml
    hbase/branches/0.90/src/docbkx/preface.xml
    hbase/branches/0.90/src/docbkx/shell.xml
    hbase/branches/0.90/src/docbkx/upgrading.xml

Modified: hbase/branches/0.90/src/docbkx/book.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.90/src/docbkx/book.xml?rev=1099696&r1=1099695&r2=1099696&view=diff
==============================================================================
--- hbase/branches/0.90/src/docbkx/book.xml (original)
+++ hbase/branches/0.90/src/docbkx/book.xml Thu May  5 06:21:55 2011
@@ -68,6 +68,7 @@
   <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="configuration.xml" />
   <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="shell.xml" />
   <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="build.xml" />
+  <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="developer.xml" />
 
 
 
@@ -124,6 +125,16 @@ throws InterruptedException, IOException
   }</programlisting>
    </para>
     </section>
+  <section xml:id="mapreduce.specex">
+  <title>Speculative Execution</title>
+  <para>It is generally advisable to turn off speculative execution for
+      MapReduce jobs that use HBase as a source.  This can either be done on a
+      per-Job basis through properties, on on the entire cluster.  Especially
+      for longer running jobs, speculative execution will create duplicate
+      map-tasks which will double-write your data to HBase; this is probably
+      not what you want.
+  </para>
+  </section>
   </chapter>
 
   <chapter xml:id="schema">
@@ -137,7 +148,7 @@ throws InterruptedException, IOException
   <title>
       Schema Creation
   </title>
-      <para>HBase schemas can be created or updated through the <link linkend="shell">HBase shell</link>
+  <para>HBase schemas can be created or updated with <xref linkend="shell" />
       or by using <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html">HBaseAdmin</link> in the Java API.
       </para>
   </section>   
@@ -186,7 +197,7 @@ throws InterruptedException, IOException
           the case described by Marc Limotte at the tail of
           <link xlink:url="https://issues.apache.org/jira/browse/HBASE-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;focusedCommentId=13005272#comment-13005272">HBASE-3551</link>
           (recommended!).
-          Therein, the indices that are kept on HBase storefiles (<link linkend="hfile">HFile</link>s)
+          Therein, the indices that are kept on HBase storefiles (<xref linkend="hfile" />)
                   to facilitate random access may end up occupyng large chunks of the HBase
                   allotted RAM because the cell value coordinates are large.
                   Mark in the above cited comment suggests upping the block size so
@@ -195,40 +206,17 @@ throws InterruptedException, IOException
                   names.
       `</para>
   </section>
-  <section xml:id="precreate.regions">
+  <section xml:id="schema.versions">
   <title>
-  Table Creation: Pre-Creating Regions
+  Number of Versions
   </title>
-<para>
-Tables in HBase are initially created with one region by default.  For bulk imports, this means that all clients will write to the same region until it is large enough to split and become distributed across the cluster.  A useful pattern to speed up the bulk import process is to pre-create empty regions.  Be somewhat conservative in this, because too-many regions can actually degrade performance.  An example of pre-creation using hex-keys is as follows (note:  this example may need to be tweaked to the individual applications keys):
-</para>
-<para>
-<programlisting>public static boolean createTable(HBaseAdmin admin, HTableDescriptor table, byte[][] splits)
-throws IOException {
-  try {
-    admin.createTable( table, splits );
-    return true;
-  } catch (TableExistsException e) {
-    logger.info("table " + table.getNameAsString() + " already exists");
-    // the table already exists...
-    return false;  
-  }
-}
-
-public static byte[][] getHexSplits(String startKey, String endKey, int numRegions) {
-  byte[][] splits = new byte[numRegions-1][];
-  BigInteger lowestKey = new BigInteger(startKey, 16);
-  BigInteger highestKey = new BigInteger(endKey, 16);
-  BigInteger range = highestKey.subtract(lowestKey);
-  BigInteger regionIncrement = range.divide(BigInteger.valueOf(numRegions));
-  lowestKey = lowestKey.add(regionIncrement);
-  for(int i=0; i &lt; numRegions-1;i++) {
-    BigInteger key = lowestKey.add(regionIncrement.multiply(BigInteger.valueOf(i)));
-    byte[] b = String.format("%016x", key).getBytes();
-    splits[i] = b;
-  }
-  return splits;
-}</programlisting>
+  <para>The number of row versions to store is configured per column
+      family via <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html">HColumnDescriptor</link>.
+      The default is 3.
+      This is an important parameter because as described in <xref linkend="datamodel" />
+      section HBase does <emphasis>not</emphasis> overwrite row values, but rather
+      stores different values per row by time (and qualifier).  Excess versions are removed during major
+      compactions.  The number of versions may need to be increased or decreased depending on application needs.
   </para>
   </section>
 
@@ -243,51 +231,57 @@ public static byte[][] getHexSplits(Stri
   </para>
   </section>
    <section xml:id="rs_metrics">
-   <title>Region Server Metrics</title>
+   <title>RegionServer Metrics</title>
           <section xml:id="hbase.regionserver.blockCacheCount"><title><varname>hbase.regionserver.blockCacheCount</varname></title>
-          <para></para>
+          <para>Block cache item count in memory.  This is the number of blocks of storefiles (HFiles) in the cache.</para>
 		  </section>
          <section xml:id="hbase.regionserver.blockCacheFree"><title><varname>hbase.regionserver.blockCacheFree</varname></title>
-          <para></para>
+          <para>Block cache memory available (bytes).</para>
 		  </section>
          <section xml:id="hbase.regionserver.blockCacheHitRatio"><title><varname>hbase.regionserver.blockCacheHitRatio</varname></title>
-          <para></para>
+          <para>Block cache hit ratio (0 to 100).  TODO:  describe impact to ratio where read requests that have cacheBlocks=false</para>
 		  </section>
           <section xml:id="hbase.regionserver.blockCacheSize"><title><varname>hbase.regionserver.blockCacheSize</varname></title>
-          <para></para>
+          <para>Block cache size in memory (bytes)</para>
+		  </section>
+          <section xml:id="hbase.regionserver.compactionQueueSize"><title><varname>hbase.regionserver.compactionQueueSize</varname></title>
+          <para>Size of the compaction queue.  This is the number of stores in the region that have been targeted for compaction.</para>
 		  </section>
           <section xml:id="hbase.regionserver.fsReadLatency_avg_time"><title><varname>hbase.regionserver.fsReadLatency_avg_time</varname></title>
-          <para></para>
+          <para>Filesystem read latency (ms).  This is the average time to read from HDFS.</para>
 		  </section>
           <section xml:id="hbase.regionserver.fsReadLatency_num_ops"><title><varname>hbase.regionserver.fsReadLatency_num_ops</varname></title>
-          <para></para>
+          <para>TODO</para>
 		  </section>
           <section xml:id="hbase.regionserver.fsSyncLatency_avg_time"><title><varname>hbase.regionserver.fsSyncLatency_avg_time</varname></title>
-          <para></para>
+          <para>Filesystem sync latency (ms)</para>
 		  </section>
           <section xml:id="hbase.regionserver.fsSyncLatency_num_ops"><title><varname>hbase.regionserver.fsSyncLatency_num_ops</varname></title>
-          <para></para>
+          <para>TODO</para>
 		  </section>
           <section xml:id="hbase.regionserver.fsWriteLatency_avg_time"><title><varname>hbase.regionserver.fsWriteLatency_avg_time</varname></title>
-          <para></para>
+          <para>Filesystem write latency (ms)</para>
 		  </section>
           <section xml:id="hbase.regionserver.fsWriteLatency_num_ops"><title><varname>hbase.regionserver.fsWriteLatency_num_ops</varname></title>
-          <para></para>
+          <para>TODO</para>
 		  </section>
           <section xml:id="hbase.regionserver.memstoreSizeMB"><title><varname>hbase.regionserver.memstoreSizeMB</varname></title>
-          <para></para>
+          <para>Sum of all the memstore sizes in this RegionServer (MB)</para>
 		  </section>
           <section xml:id="hbase.regionserver.regions"><title><varname>hbase.regionserver.regions</varname></title>
-          <para></para>
+          <para>Number of regions served by the RegionServer</para>
 		  </section>
           <section xml:id="hbase.regionserver.requests"><title><varname>hbase.regionserver.requests</varname></title>
-          <para></para>
+          <para>Total number of read and write requests.  Requests correspond to RegionServer RPC calls, thus a single Get will result in 1 request, but a Scan with caching set to 1000 will result in 1 request for each 'next' call (i.e., not each row).  A bulk-load request will constitute 1 request per HFile.</para>
 		  </section>
           <section xml:id="hbase.regionserver.storeFileIndexSizeMB"><title><varname>hbase.regionserver.storeFileIndexSizeMB</varname></title>
-          <para></para>
+          <para>Sum of all the storefile index sizes in this RegionServer (MB)</para>
 		  </section>
           <section xml:id="hbase.regionserver.stores"><title><varname>hbase.regionserver.stores</varname></title>
-          <para></para>
+          <para>Number of stores open on the RegionServer.  A store corresponds to a column family.  For example, if a table (which contains the column family) has 3 regions on a RegionServer, there will be 3 stores open for that column family. </para>
+		  </section>
+          <section xml:id="hbase.regionserver.storeFiles"><title><varname>hbase.regionserver.storeFiles</varname></title>
+          <para>Number of store filles open on the RegionServer.  A store may have more than one storefile (HFile).</para>
 		  </section>
    </section>
   </chapter>
@@ -300,11 +294,10 @@ public static byte[][] getHexSplits(Stri
 
   <chapter xml:id="datamodel">
     <title>Data Model</title>
-  <para>In short, applications store data into HBase <link linkend="table">tables</link>.
-      Tables are made of <link linkend="row">rows</link> and <emphasis>columns</emphasis>.
-      All colums in HBase belong to a particular
-      <link linkend="columnfamily">Column Family</link>.
-      Table <link linkend="cell">cells</link> -- the intersection of row and column
+    <para>In short, applications store data into an HBase table.
+        Tables are made of rows and columns.
+      All columns in HBase belong to a particular column family.
+      Table cells -- the intersection of row and column
       coordinates -- are versioned.
       A cell’s content is an uninterpreted array of bytes.
   </para>
@@ -315,6 +308,99 @@ public static byte[][] getHexSplits(Stri
       via the table row key -- its primary key.
 </para>
 
+    <section xml:id="conceptual.view"><title>Conceptual View</title>
+	<para>
+        The following example is a slightly modified form of the one on page
+        2 of the <link xlink:href="http://labs.google.com/papers/bigtable.html">BigTable</link> paper.
+    There is a table called <varname>webtable</varname> that contains two column families named
+    <varname>contents</varname> and <varname>anchor</varname>.
+    In this example, <varname>anchor</varname> contains two
+    columns (<varname>anchor:cssnsi.com</varname>, <varname>anchor:my.look.ca</varname>)
+    and <varname>contents</varname> contains one column (<varname>contents:html</varname>).
+    <note>
+        <title>Column Names</title>
+      <para>
+      By convention, a column name is made of its column family prefix and a
+      <emphasis>qualifier</emphasis>. For example, the
+      column
+      <emphasis>contents:html</emphasis> is of the column family <varname>contents</varname>
+          The colon character (<literal
+          moreinfo="none">:</literal>) delimits the column family from the
+          column family <emphasis>qualifier</emphasis>.
+    </para>
+    </note>
+    <table frame='all'><title>Table <varname>webtable</varname></title>
+	<tgroup cols='4' align='left' colsep='1' rowsep='1'>
+	<colspec colname='c1'/>
+	<colspec colname='c2'/>
+	<colspec colname='c3'/>
+	<colspec colname='c4'/>
+	<thead>
+        <row><entry>Row Key</entry><entry>Time Stamp</entry><entry>ColumnFamily <varname>contents</varname></entry><entry>ColumnFamily <varname>anchor</varname></entry></row>
+	</thead>
+	<tbody>
+        <row><entry>"com.cnn.www"</entry><entry>t9</entry><entry></entry><entry><varname>anchor:cnnsi.com</varname> = "CNN"</entry></row>
+        <row><entry>"com.cnn.www"</entry><entry>t8</entry><entry></entry><entry><varname>anchor:my.look.ca</varname> = "CNN.com"</entry></row>
+        <row><entry>"com.cnn.www"</entry><entry>t6</entry><entry><varname>contents:html</varname> = "&lt;html&gt;..."</entry><entry></entry></row>
+        <row><entry>"com.cnn.www"</entry><entry>t5</entry><entry><varname>contents:html</varname> = "&lt;html&gt;..."</entry><entry></entry></row>
+        <row><entry>"com.cnn.www"</entry><entry>t3</entry><entry><varname>contents:html</varname> = "&lt;html&gt;..."</entry><entry></entry></row>
+	</tbody>
+	</tgroup>
+	</table>
+	</para>
+	</section>
+    <section xml:id="physical.view"><title>Physical View</title>
+	<para>
+        Although at a conceptual level tables may be viewed as a sparse set of rows.
+        Physically they are stored on a per-column family basis.  New columns
+        (i.e., <varname>columnfamily:column</varname>) can be added to any
+        column family without pre-announcing them. 
+        <table frame='all'><title>ColumnFamily <varname>anchor</varname></title>
+	<tgroup cols='3' align='left' colsep='1' rowsep='1'>
+	<colspec colname='c1'/>
+	<colspec colname='c2'/>
+	<colspec colname='c3'/>
+	<thead>
+        <row><entry>Row Key</entry><entry>Time Stamp</entry><entry>Column Family <varname>anchor</varname></entry></row>
+	</thead>
+	<tbody>
+        <row><entry>"com.cnn.www"</entry><entry>t9</entry><entry><varname>anchor:cnnsi.com</varname> = "CNN"</entry></row>
+        <row><entry>"com.cnn.www"</entry><entry>t8</entry><entry><varname>anchor:my.look.ca</varname> = "CNN.com"</entry></row>
+	</tbody>
+	</tgroup>
+	</table>
+    <table frame='all'><title>ColumnFamily <varname>contents</varname></title>
+	<tgroup cols='3' align='left' colsep='1' rowsep='1'>
+	<colspec colname='c1'/>
+	<colspec colname='c2'/>
+	<colspec colname='c3'/>
+	<thead>
+	<row><entry>Row Key</entry><entry>Time Stamp</entry><entry>ColumnFamily "contents:"</entry></row>
+	</thead>
+	<tbody>
+        <row><entry>"com.cnn.www"</entry><entry>t6</entry><entry><varname>contents:html</varname> = "&lt;html&gt;..."</entry></row>
+        <row><entry>"com.cnn.www"</entry><entry>t5</entry><entry><varname>contents:html</varname> = "&lt;html&gt;..."</entry></row>
+        <row><entry>"com.cnn.www"</entry><entry>t3</entry><entry><varname>contents:html</varname> = "&lt;html&gt;..."</entry></row>
+	</tbody>
+	</tgroup>
+	</table>
+    It is important to note in the diagram above that the empty cells shown in the
+    conceptual view are not stored since they need not be in a column-oriented
+    storage format. Thus a request for the value of the <varname>contents:html</varname>
+    column at time stamp <literal>t8</literal> would return no value. Similarly, a
+    request for an <varname>anchor:my.look.ca</varname> value at time stamp
+    <literal>t9</literal> would return no value.  However, if no timestamp is
+    supplied, the most recent value for a particular column would be returned
+    and would also be the first one found since timestamps are stored in
+    descending order. Thus a request for the values of all columns in the row
+    <varname>com.cnn.www</varname> if no timestamp is specified would be:
+    the value of <varname>contents:html</varname> from time stamp
+    <literal>t6</literal>, the value of <varname>anchor:cnnsi.com</varname>
+    from time stamp <literal>t9</literal>, the value of 
+    <varname>anchor:my.look.ca</varname> from time stamp <literal>t8</literal>.
+	</para>
+	</section>
+
     <section xml:id="table">
       <title>Table</title>
       <para>
@@ -334,7 +420,7 @@ public static byte[][] getHexSplits(Stri
       <title>Column Family<indexterm><primary>Column Family</primary></indexterm></title>
         <para>
       Columns in HBase are grouped into <emphasis>column families</emphasis>.
-      All column members of a column family have a common prefix.  For example, the
+      All column members of a column family have the same prefix.  For example, the
       columns <emphasis>courses:history</emphasis> and
       <emphasis>courses:math</emphasis> are both members of the
       <emphasis>courses</emphasis> column family.
@@ -601,6 +687,66 @@ public static byte[][] getHexSplits(Stri
 
   <chapter xml:id="architecture">
     <title>Architecture</title>
+
+	<section xml:id="client">
+	 <title>Client</title>
+     <para>The HBase client
+         <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html">HTable</link>
+         is responsible for finding RegionServers that are serving the
+         particular row range of interest.  It does this by querying
+         the <code>.META.</code> and <code>-ROOT</code> catalog tables
+         (TODO: Explain).  After locating the required
+         region(s), the client <emphasis>directly</emphasis> contacts
+         the RegionServer serving that region (i.e., it does not go
+         through the master) and issues the read or write request.
+         This information is cached in the client so that subsequent requests
+         need not go through the lookup process.  Should a region be reassigned
+         either by the master load balancer or because a RegionServer has died,
+         the client will requery the catalog tables to determine the new
+         location of the user region. 
+    </para>
+    <para>Administrative functions are handled through <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html">HBaseAdmin</link>
+    </para>
+	   <section xml:id="client.connections"><title>Connections</title>
+           <para>For connection configuration information, see <xref linkend="client_dependencies" />. 
+         </para>
+         <para><link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html">HTable</link>
+instances are not thread-safe.  When creating HTable instances, it is advisable to use the same <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HBaseConfiguration">HBaseConfiguration</link>
+instance.  This will ensure sharing of ZooKeeper and socket instances to the RegionServers
+which is usually what you want.  For example, this is preferred:
+		<programlisting>HBaseConfiguration conf = HBaseConfiguration.create();
+HTable table1 = new HTable(conf, "myTable");
+HTable table2 = new HTable(conf, "myTable");</programlisting>
+		a s opposed to this:
+        <programlisting>HBaseConfiguration conf1 = HBaseConfiguration.create();
+HTable table1 = new HTable(conf1, "myTable");
+HBaseConfiguration conf2 = HBaseConfiguration.create();
+HTable table2 = new HTable(conf2, "myTable");</programlisting>
+        For more information about how connections are handled in the HBase client,
+        see <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html">HConnectionManager</link>.
+          </para>
+   	  </section>
+	   <section xml:id="client.writebuffer"><title>WriteBuffer and Batch Methods</title>
+           <para>If <xref linkend="perf.hbase.client.autoflush" /> is turned off on
+               <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html">HTable</link>,
+               <classname>Put</classname>s are sent to RegionServers when the writebuffer
+               is filled.  The writebuffer is 2MB by default.  Before an HTable instance is
+               discarded, either <methodname>close()</methodname> or
+               <methodname>flushCommits()</methodname> should be invoked so Puts
+               will not be lost. 
+	   </para> 
+       <para>For fine-grained control of batching of
+           <classname>Put</classname>s or <classname>Delete</classname>s,
+           see the <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#batch%28java.util.List%29">batch</link> methods on HTable.
+	   </para>
+	   </section>
+	   <section xml:id="client.filter"><title>Filters</title>
+           <para><link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html">Get</link> and <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html">Scan</link> instances can be
+           optionally configured with <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/Filter.html">filters</link> which are applied on the RegionServer. 
+    	   </para>
+		</section>
+	</section>
+    
     <section xml:id="daemons">
      <title>Daemons</title>
      <section xml:id="master"><title>Master</title>
@@ -650,7 +796,7 @@ public static byte[][] getHexSplits(Stri
 
         <listitem>
           <para>There is not much memory footprint difference between 1 region
-          and 10 in terms of indexes, etc, held by the regionserver.</para>
+          and 10 in terms of indexes, etc, held by the RegionServer.</para>
         </listitem>
       </itemizedlist>
 
@@ -667,7 +813,7 @@ public static byte[][] getHexSplits(Stri
         participate. The RegionServer splits a region, offlines the split
         region and then adds the daughter regions to META, opens daughters on
         the parent's hosting RegionServer and then reports the split to the
-        Master. See <link linkend="disable.splitting">Managed Splitting</link> for how to manually manage
+        Master. See <xref linkend="disable.splitting" /> for how to manually manage
         splits (and for why you might do this)</para>
       </section>
 
@@ -683,11 +829,17 @@ public static byte[][] getHexSplits(Stri
 
       <section xml:id="store">
           <title>Store</title>
-          <para>A Store hosts a MemStore and 0 or more StoreFiles.
-              StoreFiles are HFiles.
+          <para>A Store hosts a MemStore and 0 or more StoreFiles (HFiles). A Store corresponds to a column family for a table for a given region.
           </para>
+    <section xml:id="store.memstore">
+      <title>MemStore</title>
+      <para>The MemStore holds in-memory modifications to the Store.  Modifications are KeyValues.
+       When asked to flush, current memstore is moved to snapshot and is cleared. 
+       HBase continues to serve edits out of new memstore and backing snapshot until flusher reports in that the 
+       flush succeeded. At this point the snapshot is let go.</para>
+      </section>
     <section xml:id="hfile">
-      <title>HFile</title>
+      <title>StoreFile (HFile)</title>
       <section><title>HFile Format</title>
           <para>The <emphasis>hfile</emphasis> file format is based on
               the SSTable file described in the <link xlink:href="http://labs.google.com/papers/bigtable.html">BigTable [2006]</link> paper and on
@@ -712,42 +864,55 @@ public static byte[][] getHexSplits(Stri
         tool.</para>
       </section>
       </section>
+      <section xml:id="compaction">
+        <title>Compaction</title>
+        <para>There are two types of compactions:  minor and major.  Minor compactions will usually pick up a couple of the smaller adjacent
+         files and rewrite them as one.  Minors do not drop deletes or expired cells, only major compactions do this.  Sometimes a minor compaction
+         will pick up all  the files in the store and in this case it actually promotes itself to being a major compaction.  
+         For a description of how a minor compaction picks files to compact, see the <link xlink:href="http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/Store.html#836">ascii diagram in the Store source code.</link>
+         </para>
+         <para>After a major compaction runs there will be a single storefile per store, and this will help performance usually.  Caution:  major compactions rewrite all of the stores data and on a loaded system, this may not be tenable;
+             major compactions will usually have to be <xref linkend="disable.splitting" /> on large systems.
+        </para>
       </section>
 
-    </section>
-  </chapter>
-
-  <chapter xml:id="wal">
-    <title >The WAL</title>
-
-    <subtitle>HBase's<link
-    xlink:href="http://en.wikipedia.org/wiki/Write-ahead_logging"> Write-Ahead
-    Log</link></subtitle>
 
-    <para>Each RegionServer adds updates to its Write-ahead Log (WAL)
-    first, and then to memory.</para>
+      </section>
 
-    <section xml:id="purpose.wal">
-      <title>What is the purpose of the HBase WAL</title>
 
-      <para>
-     See the Wikipedia
-     <link xlink:href="http://en.wikipedia.org/wiki/Write-ahead_logging">Write-Ahead
-    Log</link> article.
-
-      </para>
-    </section>
+  </section>
+      <section xml:id="wal">
+       <title >Write Ahead Log (WAL)</title>
 
-    <section xml:id="wal_splitting">
-      <title>WAL splitting</title>
+       <section xml:id="purpose.wal">
+         <title>Purpose</title>
 
-      <subtitle>How edits are recovered from a crashed RegionServer</subtitle>
+        <para>Each RegionServer adds updates (Puts, Deletes) to its write-ahead log (WAL)
+            first, and then to the <xref linkend="store.memstore"/> for the affected <xref linkend="store" />.  
+        This ensures that HBase has durable writes. Without WAL, there is the possibility of data loss in the case of a RegionServer failure 
+        before each MemStore is flushed and new StoreFiles are written.  <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/wal/HLog.html">HLog</link> 
+        is the HBase WAL implementation, and there is one HLog instance per RegionServer.
+       </para>The WAL is in HDFS in <filename>/hbase/.logs/</filename> with subdirectories per region.
+       <para>
+        For more general information about the concept of write ahead logs, see the Wikipedia
+        <link xlink:href="http://en.wikipedia.org/wiki/Write-ahead_logging">Write-Ahead Log</link> article.
+       </para>
+       </section>
+       <section xml:id="wal_flush">
+        <title>WAL Flushing</title>
+          <para>TODO (describe).
+          </para>
+        </section>
 
-      <para>When a RegionServer crashes, it will lose its ephemeral lease in
-      ZooKeeper...TODO</para>
+        <section xml:id="wal_splitting">
+         <title>WAL Splitting</title>
 
-      <section>
-        <title><varname>hbase.hlog.split.skip.errors</varname></title>
+        <section><title>How edits are recovered from a crashed RegionServer</title>
+         <para>When a RegionServer crashes, it will lose its ephemeral lease in
+         ZooKeeper...TODO</para>
+		 </section>
+         <section>
+         <title><varname>hbase.hlog.split.skip.errors</varname></title>
 
         <para>When set to <constant>true</constant>, the default, any error
         encountered splitting will be logged, the problematic WAL will be
@@ -780,8 +945,9 @@ public static byte[][] getHexSplits(Stri
       </section>
     </section>
 
+    </section>
   </chapter>
-
+  
   <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="performance.xml" />
 
   <chapter xml:id="blooms">
@@ -903,6 +1069,8 @@ public static byte[][] getHexSplits(Stri
     </section>
   </chapter>
 
+  <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="troubleshooting.xml" />
+
   <appendix xml:id="tools">
     <title >Tools</title>
 
@@ -924,7 +1092,7 @@ public static byte[][] getHexSplits(Stri
         </para>
     </section>
     <section><title>HFile Tool</title>
-        <para>See <link linkend="hfile_tool" >HFile Tool</link>.</para>
+        <para>See <xref linkend="hfile_tool" />.</para>
     </section>
     <section xml:id="wal_tools">
       <title>WAL Tools</title>
@@ -947,7 +1115,136 @@ public static byte[][] getHexSplits(Stri
       </section>
     </section>
     <section xml:id="compression.tool"><title>Compression Tool</title>
-        <para>See <link linkend="compression.tool" >Compression Tool</link>.</para>
+        <para>See <xref linkend="compression.tool" />.</para>
+    </section>
+    <section xml:id="decommission"><title>Node Decommission</title>
+        <para>You can stop an individual RegionServer by running the following
+            script in the HBase directory on the particular  node:
+            <programlisting>$ ./bin/hbase-daemon.sh stop regionserver</programlisting>
+            The RegionServer will first close all regions and then shut itself down.
+            On shutdown, the RegionServer's ephemeral node in ZooKeeper will expire.
+            The master will notice the RegionServer gone and will treat it as
+            a 'crashed' server; it will reassign the nodes the RegionServer was carrying.
+            <note><title>Disable the Load Balancer before Decommissioning a node</title>
+             <para>If the load balancer runs while a node is shutting down, then
+                 there could be contention between the Load Balancer and the
+                 Master's recovery of the just decommissioned RegionServer.
+                 Avoid any problems by disabling the balancer first.
+                 See <xref linkend="lb" /> below.
+             </para>
+            </note>
+        </para>
+        <para>
+        A downside to the above stop of a RegionServer is that regions could be offline for
+        a good period of time.  Regions are closed in order.  If many regions on the server, the
+        first region to close may not be back online until all regions close and after the master
+        notices the RegionServer's znode gone.  In HBase 0.90.2, we added facility for having
+        a node gradually shed its load and then shutdown itself down.  HBase 0.90.2 added the
+            <filename>graceful_stop.sh</filename> script.  Here is its usage:
+            <programlisting>$ ./bin/graceful_stop.sh 
+Usage: graceful_stop.sh [--config &amp;conf-dir>] [--restart] [--reload] [--thrift] [--rest] &amp;hostname>
+ thrift      If we should stop/start thrift before/after the hbase stop/start
+ rest        If we should stop/start rest before/after the hbase stop/start
+ restart     If we should restart after graceful stop
+ reload      Move offloaded regions back on to the stopped server
+ debug       Move offloaded regions back on to the stopped server
+ hostname    Hostname of server we are to stop</programlisting>
+        </para>
+        <para>
+            To decommission a loaded RegionServer, run the following:
+            <programlisting>$ ./bin/graceful_stop.sh HOSTNAME</programlisting>
+            where <varname>HOSTNAME</varname> is the host carrying the RegionServer
+            you would decommission.  
+            <note><title>On <varname>HOSTNAME</varname></title>
+                <para>The <varname>HOSTNAME</varname> passed to <filename>graceful_stop.sh</filename>
+            must match the hostname that hbase is using to identify RegionServers.
+            Check the list of RegionServers in the master UI for how HBase is
+            referring to servers. Its usually hostname but can also be FQDN.
+            Whatever HBase is using, this is what you should pass the
+            <filename>graceful_stop.sh</filename> decommission
+            script.  If you pass IPs, the script is not yet smart enough to make
+            a hostname (or FQDN) of it and so it will fail when it checks if server is
+            currently running; the graceful unloading of regions will not run.
+            </para>
+        </note> The <filename>graceful_stop.sh</filename> script will move the regions off the
+            decommissioned RegionServer one at a time to minimize region churn.
+            It will verify the region deployed in the new location before it
+            will moves the next region and so on until the decommissioned server
+            is carrying zero regions.  At this point, the <filename>graceful_stop.sh</filename>
+            tells the RegionServer <command>stop</command>.  The master will at this point notice the
+            RegionServer gone but all regions will have already been redeployed
+            and because the RegionServer went down cleanly, there will be no
+            WAL logs to split.
+            <note xml:id="lb"><title>Load Balancer</title>
+            <para> 
+                It is assumed that the Region Load Balancer is disabled while the
+                <command>graceful_stop</command> script runs (otherwise the balancer
+                and the decommission script will end up fighting over region deployments).
+                Use the shell to disable the balancer:
+                <programlisting>hbase(main):001:0> balance_switch false
+true
+0 row(s) in 0.3590 seconds</programlisting>
+This turns the balancer OFF.  To reenable, do:
+                <programlisting>hbase(main):001:0> balance_switch true
+false
+0 row(s) in 0.3590 seconds</programlisting>
+            </para> 
+        </note>
+        </para>
+        <section xml:id="rolling">
+            <title>Rolling Restart</title>
+        <para>
+            You can also ask this script to restart a RegionServer after the shutdown
+            AND move its old regions back into place.  The latter you might do to
+            retain data locality.  A primitive rolling restart might be effected by
+            running something like the following:
+            <programlisting>$ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart --reload --debug $i; done &amp;> /tmp/log.txt &amp;
+            </programlisting>
+            Tail the output of <filename>/tmp/log.txt</filename> to follow the scripts
+            progress. The above does RegionServers only.  Be sure to disable the
+            load balancer before doing the above.  You'd need to do the master
+            update separately.  Do it before you run the above script.
+            Here is a pseudo-script for how you might craft a rolling restart script:
+            <orderedlist>
+                <listitem><para>Untar your release, make sure of its configuration and
+                        then rsync it across the cluster. If this is 0.90.2, patch it
+                        with HBASE-3744 and HBASE-3756.
+                    </para>
+                </listitem>
+                <listitem>
+                    <para>Run hbck to ensure the cluster consistent
+                        <programlisting>$ ./bin/hbase hbck</programlisting>
+                    Effect repairs if inconsistent.
+                    </para>
+                </listitem>
+                <listitem>
+                    <para>Restart the Master: <programlisting>$ ./bin/hbase-daemon.sh stop master; ./bin/hbase-daemon.sh start master</programlisting>
+                    </para>
+                </listitem>
+                <listitem>
+                    <para>
+                       Disable the region balancer:<programlisting>$ echo "balance_switch false" | ./bin/hbase</programlisting>
+                    </para>
+                </listitem>
+                <listitem>
+                     <para>Run the <filename>graceful_stop.sh</filename> script per RegionServer.  For example:
+            <programlisting>$ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart --reload --debug $i; done &amp;> /tmp/log.txt &amp;
+            </programlisting>
+                     If you are running thrift or rest servers on the RegionServer, pass --thrift or --rest options (See usage
+                     for <filename>graceful_stop.sh</filename> script).
+                 </para>
+                </listitem>
+                <listitem>
+                    <para>Restart the Master again.  This will clear out dead servers list and reenable the balancer.
+                    </para>
+                </listitem>
+                <listitem>
+                    <para>Run hbck to ensure the cluster is consistent.
+                    </para>
+                </listitem>
+            </orderedlist>
+        </para>
+    </section>
     </section>
   </appendix>
 
@@ -996,7 +1293,7 @@ public static byte[][] getHexSplits(Stri
     LZO
     </title>
     <para>
-    See <link linkend="lzo">LZO Compression</link> above.
+        See <xref linkend="lzo" /> above.
     </para>
     </section>
 
@@ -1011,7 +1308,7 @@ public static byte[][] getHexSplits(Stri
     available on the CLASSPATH; in this case it will use native
     compressors instead (If the native libs are NOT present,
     you will see lots of <emphasis>Got brand-new compressor</emphasis>
-    reports in your logs; see <link linkend="brand.new.compressor">FAQ</link>).
+    reports in your logs; see <xref linkend="brand.new.compressor" />).
     </para>
     </section>
   </appendix>
@@ -1025,8 +1322,27 @@ public static byte[][] getHexSplits(Stri
             <answer>
                 <para>
               See the FAQ that is up on the wiki, <link xlink:href="http://wiki.apache.org/hadoop/Hbase/FAQ">HBase Wiki FAQ</link>
-              as well as the <link xlink:href="http://wiki.apache.org/hadoop/Hbase/Troubleshooting">Troubleshooting</link> page and
-              the <link xlink:href="http://wiki.apache.org/hadoop/Hbase/FrequentlySeenErrors">Frequently Seen Errors</link> page.
+              as well as the <link xlink:href="http://wiki.apache.org/hadoop/Hbase/Troubleshooting">Troubleshooting</link> page.
+                </para>
+            </answer>
+        </qandaentry>
+        <qandaentry xml:id="faq.sql">
+            <question><para>Does HBase support SQL?</para></question>
+            <answer>
+                <para>
+                    Not really.  SQL-ish support for HBase via <link xlink:href="http://hive.apache.org/">Hive</link> is in development, however Hive is based on MapReduce which is not generally suitable for low-latency requests.
+                    See the <xref linkend="datamodel" /> section for examples on the HBase client.
+                </para>
+            </answer>
+        </qandaentry>
+        <qandaentry xml:id="faq.hdfs.hbase">
+            <question><para>How does HBase work on top of HDFS?</para></question>
+            <answer>
+                <para>
+                    <link xlink:href="http://hadoop.apache.org/hdfs/">HDFS</link> is a distributed file system that is well suited for the storage of large files.  It's documentation 
+                    states that it is not, however, a general purpose file system, and does not provide fast individual record lookups in files. 
+                    HBase, on the other hand, is built on top of HDFS and provides fast record lookups (and updates) for large tables.  This can sometimes be a point of conceptual confusion.
+                    See the <xref linkend="datamodel" /> and <xref linkend="architecture" /> sections for more information on how HBase achieves its goals.
                 </para>
             </answer>
         </qandaentry>
@@ -1093,6 +1409,18 @@ When I build, why do I always get <code>
                 </para>
             </answer>
         </qandaentry>
+        <qandaentry>
+            <question><para>
+                    Why do I see double the actual count of regionservers, once by domain name and then by IP?
+            </para></question>
+            <answer>
+                <para>
+                    Fix your DNS.  In versions of HBase before 0.92.x, reverse DNS needs to give same answer
+                    as forward lookup. See <link xlink:href="https://issues.apache.org/jira/browse/HBASE-3431">HBASE 3431
+                        Regionserver is not using the name given it by the master; double entry in master listing of servers</link> for gorey details.
+                </para>
+            </answer>
+        </qandaentry>
     </qandadiv>
     <qandadiv><title>How do I...?</title>
         <qandaentry xml:id="secondary.indices">
@@ -1112,6 +1440,7 @@ When I build, why do I always get <code>
 
   <appendix>
       <title xml:id="ycsb"><link xlink:href="https://github.com/brianfrankcooper/YCSB/">YCSB: The Yahoo! Cloud Serving Benchmark</link> and HBase</title>
+      <para>TODO: Describe how YCSB is poor for putting up a decent cluster load.</para>
       <para>TODO: Describe setup of YCSB for HBase</para>
       <para>Ted Dunning redid YCSB so its mavenized and added facility for verifying workloads.  See <link xlink:href="https://github.com/tdunning/YCSB">Ted Dunning's YCSB</link>.</para>
 

Modified: hbase/branches/0.90/src/docbkx/build.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.90/src/docbkx/build.xml?rev=1099696&r1=1099695&r2=1099696&view=diff
==============================================================================
--- hbase/branches/0.90/src/docbkx/build.xml (original)
+++ hbase/branches/0.90/src/docbkx/build.xml Thu May  5 06:21:55 2011
@@ -10,11 +10,34 @@
     <title>Building HBase</title>
     <section xml:id="mvn_repo">
         <title>Adding an HBase release to Apache's Maven Repository</title>
-        <para>Follow the instructions at <link xlink:href="http://www.apache.org/dev/publishing-maven-artifacts.html">Publishing Maven Artifacts</link>.
+        <para>Follow the instructions at
+        <link xlink:href="http://www.apache.org/dev/publishing-maven-artifacts.html">Publishing Maven Artifacts</link>.
             The 'trick' to making it all work is answering the questions put to you by the mvn release plugin properly,
-            making sure it is using the actual branch,
-            and finally, before doing the mvn release:perform,
-            VERY IMPORTANT, hand edit the release.properties file that was put under HBASE_HOME by the previous step, release:perform. You need to edit it to make it point at right locations in SVN.
+            making sure it is using the actual branch AND before doing the <command>mvn release:perform</command> step,
+            VERY IMPORTANT, hand edit the release.properties file that was put under <varname>${HBASE_HOME}</varname>
+            by the previous step, <command>release:perform</command>. You need to edit it to make it point at
+            right locations in SVN.
+        </para>
+        <para>If you see run into the below, its because you need to edit version in the pom.xml and add
+        <code>-SNAPSHOT</code> to the version (and commit).
+        <programlisting>[INFO] Scanning for projects...
+[INFO] Searching repository for plugin with prefix: 'release'.
+[INFO] ------------------------------------------------------------------------
+[INFO] Building HBase
+[INFO]    task-segment: [release:prepare] (aggregator-style)
+[INFO] ------------------------------------------------------------------------
+[INFO] [release:prepare {execution: default-cli}]
+[INFO] ------------------------------------------------------------------------
+[ERROR] BUILD FAILURE
+[INFO] ------------------------------------------------------------------------
+[INFO] You don't have a SNAPSHOT project in the reactor projects list.
+[INFO] ------------------------------------------------------------------------
+[INFO] For more information, run Maven with the -e switch
+[INFO] ------------------------------------------------------------------------
+[INFO] Total time: 3 seconds
+[INFO] Finished at: Sat Mar 26 18:11:07 PDT 2011
+[INFO] Final Memory: 35M/423M
+[INFO] -----------------------------------------------------------------------</programlisting>
         </para>
     </section>
           

Modified: hbase/branches/0.90/src/docbkx/configuration.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.90/src/docbkx/configuration.xml?rev=1099696&r1=1099695&r2=1099696&view=diff
==============================================================================
--- hbase/branches/0.90/src/docbkx/configuration.xml (original)
+++ hbase/branches/0.90/src/docbkx/configuration.xml Thu May  5 06:21:55 2011
@@ -40,7 +40,7 @@ to ensure well-formedness of your docume
     for HBase, site specific customizations go into
     the file <filename>conf/hbase-site.xml</filename>.
     For the list of configurable properties, see
-    <link linkend="hbase_default_configurations">Default HBase Configurations</link>
+    <xref linkend="hbase_default_configurations" />
     below or view the raw <filename>hbase-default.xml</filename>
     source file in the HBase source code at
     <filename>src/main/resources</filename>.
@@ -99,10 +99,10 @@ to ensure well-formedness of your docume
 
 
       <section xml:id="required_configuration"><title>Required Configurations</title>
-      <para>See the <link linkend="requirements">Requirements</link> section.
+          <para>See <xref linkend="requirements" />.
       It lists at least two required configurations needed running HBase bearing
-      load: i.e. <link linkend="ulimit">file descriptors <varname>ulimit</varname></link> and
-      <link linkend="dfs.datanode.max.xcievers"><varname>dfs.datanode.max.xcievers</varname></link>.
+      load: i.e. <xref linkend="ulimit" /> and
+      <xref linkend="dfs.datanode.max.xcievers" />.
       </para>
       </section>
 
@@ -114,7 +114,7 @@ to ensure well-formedness of your docume
               a minute or even less so the Master notices failures the sooner.
               Before changing this value, be sure you have your JVM garbage collection
               configuration under control otherwise, a long garbage collection that lasts
-              beyond the zookeeper session timeout will take out
+              beyond the ZooKeeper session timeout will take out
               your RegionServer (You might be fine with this -- you probably want recovery to start
           on the server if a RegionServer has been in GC for a long period of time).</para> 
 
@@ -185,10 +185,10 @@ to ensure well-formedness of your docume
       fixup on the new machine.  In versions since HBase 0.90.0, we should
       fail in a way that makes it plain what the problem is, but maybe not.
       Remember you read this paragraph<footnote><para>See
-      <link linkend="hbase.regionserver.codecs">hbase.regionserver.codecs</link>
+              <xref linkend="hbase.regionserver.codecs" />
       for a feature to help protect against failed LZO install</para></footnote>.
       </para>
-      <para>See also the <link linkend="compression">Compression Appendix</link>
+      <para>See also <xref linkend="compression" />
       at the tail of this book.</para>
       </section>
       <section xml:id="bigger.regions">
@@ -274,7 +274,7 @@ of all regions.
       </para>
       <para>
           Minimally, a client of HBase needs the hbase, hadoop, log4j, commons-logging, commons-lang,
-          and zookeeper jars in its <varname>CLASSPATH</varname> connecting to a cluster.
+          and ZooKeeper jars in its <varname>CLASSPATH</varname> connecting to a cluster.
       </para>
         <para>
           An example basic <filename>hbase-site.xml</filename> for client only
@@ -303,11 +303,11 @@ of all regions.
         (Invocation will also factor in any <filename>hbase-default.xml</filename> found;
         an hbase-default.xml ships inside the <filename>hbase.X.X.X.jar</filename>). 
         It is also possible to specify configuration directly without having to read from a
-        <filename>hbase-site.xml</filename>.  For example, to set the
-        <link linkend="zookeeper">zookeeper</link> ensemble for the cluster programmatically do as follows:
+        <filename>hbase-site.xml</filename>.  For example, to set the ZooKeeper
+        ensemble for the cluster programmatically do as follows:
         <programlisting>Configuration config = HBaseConfiguration.create();
 config.set("hbase.zookeeper.quorum", "localhost");  // Here we are running zookeeper locally</programlisting>    
-        If multiple <link linkend="zookeeper">zookeeper</link> instances make up your zookeeper ensemble,
+        If multiple ZooKeeper instances make up your ZooKeeper ensemble,
         they may be specified in a comma-separated list (just as in the <filename>hbase-site.xml</filename> file).
         This populated <classname>Configuration</classname> instance can then be passed to an 
         <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html">HTable</link>,

Modified: hbase/branches/0.90/src/docbkx/getting_started.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.90/src/docbkx/getting_started.xml?rev=1099696&r1=1099695&r2=1099696&view=diff
==============================================================================
--- hbase/branches/0.90/src/docbkx/getting_started.xml (original)
+++ hbase/branches/0.90/src/docbkx/getting_started.xml Thu May  5 06:21:55 2011
@@ -12,9 +12,9 @@
   <section>
     <title>Introduction</title>
 
-    <para><link linkend="quickstart">Quick Start</link> will get you up and
+    <para><xref linkend="quickstart" /> will get you up and
     running on a single-node instance of HBase using the local filesystem. The
-    <link linkend="notsoquick">Not-so-quick Start Guide</link> describes setup
+    <xref linkend="notsoquick" /> describes setup
     of HBase in distributed mode running on top of HDFS.</para>
   </section>
 
@@ -23,7 +23,7 @@
 
     <para>This guide describes setup of a standalone HBase instance that uses
     the local filesystem. It leads you through creating a table, inserting
-    rows via the <link linkend="shell">HBase Shell</link>, and then cleaning
+    rows via the HBase <command>shell</command>, and then cleaning
     up and shutting down your standalone HBase instance. The below exercise
     should take no more than ten minutes (not including download time).</para>
 
@@ -97,8 +97,7 @@ starting Master, logging to logs/hbase-u
     <section xml:id="shell_exercises">
       <title>Shell Exercises</title>
 
-      <para>Connect to your running HBase via the <link linkend="shell">HBase
-      Shell</link>.</para>
+      <para>Connect to your running HBase via the <command>shell</command>.</para>
 
       <para><programlisting>$ ./bin/hbase shell
 HBase Shell; enter 'help&lt;RETURN&gt;' for list of supported commands.
@@ -114,8 +113,7 @@ hbase(main):001:0&gt; </programlisting><
       HBase shell; in particular note how table names, rows, and columns,
       etc., must be quoted.</para>
 
-      <para>Create a table named <varname>test</varname> with a single <link
-      linkend="columnfamily">column family</link> named <varname>cf</varname>.
+  <para>Create a table named <varname>test</varname> with a single column family named <varname>cf</varname>.
       Verify its creation by listing all tables and then insert some
       values.</para>
 
@@ -133,8 +131,7 @@ hbase(main):006:0&gt; put 'test', 'row3'
 
       <para>Above we inserted 3 values, one at a time. The first insert is at
       <varname>row1</varname>, column <varname>cf:a</varname> with a value of
-      <varname>value1</varname>. Columns in HBase are comprised of a <link
-      linkend="columnfamily">column family</link> prefix --
+      <varname>value1</varname>. Columns in HBase are comprised of a column family prefix --
       <varname>cf</varname> in this example -- followed by a colon and then a
       column qualifier suffix (<varname>a</varname> in this case).</para>
 
@@ -182,8 +179,7 @@ stopping hbase...............</programli
       <title>Where to go next</title>
 
       <para>The above described standalone setup is good for testing and
-      experiments only. Move on to the next section, the <link
-      linkend="notsoquick">Not-so-quick Start Guide</link> where we'll go into
+          experiments only. Next move on to <xref linkend="notsoquick" /> where we'll go into
       depth on the different HBase run modes, requirements and critical
       configurations needed setting up a distributed HBase deploy.</para>
     </section>
@@ -221,19 +217,17 @@ stopping hbase...............</programli
         will lose data unless it is running on an HDFS that has a durable
         <code>sync</code>. Currently only the <link
         xlink:href="http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/">branch-0.20-append</link>
-        branch has this attribute <footnote>
+        branch has this attribute<footnote>
             <para>See <link
             xlink:href="http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/CHANGES.txt">CHANGES.txt</link>
             in branch-0.20-append to see list of patches involved adding
             append on the Hadoop 0.20 branch.</para>
           </footnote>. No official releases have been made from this branch up
         to now so you will have to build your own Hadoop from the tip of this
-        branch. Check it out using this url, <link
-        xlink:href="https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append/">branch-0.20-append</link>.
-        Scroll down in the Hadoop <link
-        xlink:href="http://wiki.apache.org/hadoop/HowToRelease">How To
-        Release</link> to the section <emphasis>Build Requirements</emphasis>
-        for instruction on how to build Hadoop.</para>
+        branch.  Michael Noll has written a detailed blog,
+        <link xlink:href="http://www.michael-noll.com/blog/2011/04/14/building-an-hadoop-0-20-x-version-for-hbase-0-90-2/">Building
+        an Hadoop 0.20.x version for HBase 0.90.2</link>, on how to build an
+        Hadoop from branch-0.20-append.  Recommended.</para>
 
         <para>Or rather than build your own, you could use Cloudera's <link
         xlink:href="http://archive.cloudera.com/docs/">CDH3</link>. CDH has
@@ -319,23 +313,39 @@ stopping hbase...............</programli
       </section>
 
       <section xml:id="ulimit">
-        <title><varname>ulimit</varname><indexterm>
+        <title>
+          <varname>ulimit</varname><indexterm>
             <primary>ulimit</primary>
-          </indexterm></title>
-
-        <para>HBase is a database, it uses a lot of files at the same time.
-        The default ulimit -n of 1024 on *nix systems is insufficient. Any
-        significant amount of loading will lead you to <link
-        xlink:href="http://wiki.apache.org/hadoop/Hbase/FAQ#A6">FAQ: Why do I
+            </indexterm>
+            and
+          <varname>nproc</varname><indexterm>
+            <primary>nproc</primary>
+            </indexterm>
+        </title>
+
+        <para>HBase is a database.  It uses a lot of files all at the same time.
+        The default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems
+        is insufficient (On mac os x its 256). Any significant amount of loading will
+        lead you to <link xlink:href="http://wiki.apache.org/hadoop/Hbase/FAQ#A6">FAQ: Why do I
         see "java.io.IOException...(Too many open files)" in my logs?</link>.
         You may also notice errors such as <programlisting>
       2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
       2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901
       </programlisting> Do yourself a favor and change the upper bound on the
         number of file descriptors. Set it to north of 10k. See the above
-        referenced FAQ for how.</para>
+        referenced FAQ for how.  You should also up the hbase users'
+        <varname>nproc</varname> setting; under load, a low-nproc
+        setting could manifest as <classname>OutOfMemoryError</classname>
+        <footnote><para>See Jack Levin's <link xlink:href="">major hdfs issues</link>
+                note up on the user list.</para></footnote>
+        <footnote><para>The requirement that a database requires upping of system limits
+        is not peculiar to HBase.  See for example the section
+        <emphasis>Setting Shell Limits for the Oracle User</emphasis> in
+        <link xlink:href="http://www.akadia.com/services/ora_linux_install_10g.html">
+        Short Guide to install Oracle 10 on Linux</link>.</para></footnote>.
+    </para>
 
-        <para>To be clear, upping the file descriptors for the user who is
+        <para>To be clear, upping the file descriptors and nproc for the user who is
         running the HBase process is an operating system configuration, not an
         HBase configuration. Also, a common mistake is that administrators
         will up the file descriptors for a particular user but for whatever
@@ -358,12 +368,12 @@ stopping hbase...............</programli
           a line like: <programlisting>hadoop  -       nofile  32768</programlisting>
           Replace <varname>hadoop</varname> with whatever user is running
           Hadoop and HBase. If you have separate users, you will need 2
-          entries, one for each user.</para>
+          entries, one for each user.  In the same file set nproc hard and soft
+          limits.  For example: <programlisting>hadoop soft/hard nproc 32000</programlisting>.</para>
 
           <para>In the file <filename>/etc/pam.d/common-session</filename> add
           as the last line in the file: <programlisting>session required  pam_limits.so</programlisting>
-          Otherwise the changes in
-          <filename>/etc/security/limits.conf</filename> won't be
+          Otherwise the changes in <filename>/etc/security/limits.conf</filename> won't be
           applied.</para>
 
           <para>Don't forget to log out and back in again for the changes to
@@ -399,7 +409,8 @@ stopping hbase...............</programli
         <code>10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block
         blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node:
         java.io.IOException: No live nodes contain current block. Will get new
-        block locations from namenode and retry...</code></para>
+        block locations from namenode and retry...</code>
+        <footnote><para>See <link xlink:href="http://ccgtech.blogspot.com/2010/02/hadoop-hdfs-deceived-by-xciever.html">Hadoop HDFS: Deceived by Xciever</link> for an informative rant on xceivering.</para></footnote></para>
       </section>
 
       <section xml:id="windows">
@@ -422,9 +433,7 @@ stopping hbase...............</programli
     <section xml:id="standalone_dist">
       <title>HBase run modes: Standalone and Distributed</title>
 
-      <para>HBase has two run modes: <link
-      linkend="standalone">standalone</link> and <link
-      linkend="distributed">distributed</link>. Out of the box, HBase runs in
+      <para>HBase has two run modes: <xref linkend="standalone" /> and <xref linkend="distributed" />. Out of the box, HBase runs in
       standalone mode. To set up a distributed deploy, you will need to
       configure HBase by editing files in the HBase <filename>conf</filename>
       directory.</para>
@@ -441,10 +450,10 @@ stopping hbase...............</programli
         <title>Standalone HBase</title>
 
         <para>This is the default mode. Standalone mode is what is described
-        in the <link linkend="quickstart">quickstart</link> section. In
+            in the <xref linkend="quickstart" /> section. In
         standalone mode, HBase does not use HDFS -- it uses the local
         filesystem instead -- and it runs all HBase daemons and a local
-        zookeeper all up in the same JVM. Zookeeper binds to a well known port
+        ZooKeeper all up in the same JVM. Zookeeper binds to a well known port
         so clients may talk to HBase.</para>
       </section>
 
@@ -470,8 +479,7 @@ stopping hbase...............</programli
         verification and exploration of your install, whether a
         <emphasis>pseudo-distributed</emphasis> or
         <emphasis>fully-distributed</emphasis> configuration is described in a
-        section that follows, <link linkend="confirm">Running and Confirming
-        your Installation</link>. The same verification script applies to both
+        section that follows, <xref linkend="confirm" />. The same verification script applies to both
         deploy types.</para>
 
         <section xml:id="pseudo">
@@ -484,10 +492,8 @@ stopping hbase...............</programli
 
           <para>Once you have confirmed your HDFS setup, edit
           <filename>conf/hbase-site.xml</filename>. This is the file into
-          which you add local customizations and overrides for <link
-          linkend="hbase_default_configurations">Default HBase
-          Configurations</link> and <link linkend="hdfs_client_conf">HDFS
-          Client Configurations</link>. Point HBase at the running Hadoop HDFS
+          which you add local customizations and overrides for
+          <xreg linkend="hbase_default_configurations" /> and <xref linkend="hdfs_client_conf" />. Point HBase at the running Hadoop HDFS
           instance by setting the <varname>hbase.rootdir</varname> property.
           This property points HBase at the Hadoop filesystem instance to use.
           For example, adding the properties below to your
@@ -502,7 +508,7 @@ stopping hbase...............</programli
   &lt;property&gt;
     &lt;name&gt;hbase.rootdir&lt;/name&gt;
     &lt;value&gt;hdfs://localhost:9000/hbase&lt;/value&gt;
-    &lt;description&gt;The directory shared by region servers.
+    &lt;description&gt;The directory shared by RegionServers.
     &lt;/description&gt;
   &lt;/property&gt;
   &lt;property&gt;
@@ -528,13 +534,12 @@ stopping hbase...............</programli
             want to connect from a remote location.</para>
           </note>
 
-          <para>Now skip to <link linkend="confirm">Running and Confirming
-          your Installation</link> for how to start and verify your
+          <para>Now skip to <xref linkend="confirm" /> for how to start and verify your
           pseudo-distributed install. <footnote>
               <para>See <link
               xlink:href="http://hbase.apache.org/pseudo-distributed.html">Pseudo-distributed
               mode extras</link> for notes on how to start extra Masters and
-              regionservers when running pseudo-distributed.</para>
+              RegionServers when running pseudo-distributed.</para>
             </footnote></para>
         </section>
 
@@ -559,7 +564,7 @@ stopping hbase...............</programli
   &lt;property&gt;
     &lt;name&gt;hbase.rootdir&lt;/name&gt;
     &lt;value&gt;hdfs://namenode.example.org:9000/hbase&lt;/value&gt;
-    &lt;description&gt;The directory shared by region servers.
+    &lt;description&gt;The directory shared by RegionServers.
     &lt;/description&gt;
   &lt;/property&gt;
   &lt;property&gt;
@@ -579,8 +584,7 @@ stopping hbase...............</programli
 
             <para>In addition, a fully-distributed mode requires that you
             modify <filename>conf/regionservers</filename>. The
-            <filename><link
-            linkend="regionservrers">regionservers</link></filename> file
+            <xref linkend="regionservers" /> file
             lists all hosts that you would have running
             <application>HRegionServer</application>s, one host per line (This
             file in HBase is like the Hadoop <filename>slaves</filename>
@@ -619,9 +623,7 @@ stopping hbase...............</programli
             by setting the
             <varname>hbase.zookeeper.property.clientPort</varname> property.
             For all default values used by HBase, including ZooKeeper
-            configuration, see the section <link
-            linkend="hbase_default_configurations">Default HBase
-            Configurations</link>. Look for the
+            configuration, see <xref linkend="hbase_default_configurations" />. Look for the
             <varname>hbase.zookeeper.property</varname> prefix <footnote>
                 <para>For the full list of ZooKeeper configurations, see
                 ZooKeeper's <filename>zoo.cfg</filename>. HBase does not ship
@@ -820,8 +822,7 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {star
 
          
 
-        <para>Once HBase has started, see the <link
-        linkend="shell_exercises">Shell Exercises</link> section for how to
+    <para>Once HBase has started, see the <xref linkend="shell_exercises" /> for how to
         create tables, add data, scan your insertions, and finally disable and
         drop your tables.</para>
 
@@ -872,7 +873,7 @@ stopping hbase...............</programli
   &lt;property&gt;
     &lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
     &lt;value&gt;example1,example2,example3&lt;/value&gt;
-    &lt;description&gt;The directory shared by region servers.
+    &lt;description&gt;The directory shared by RegionServers.
     &lt;/description&gt;
   &lt;/property&gt;
   &lt;property&gt;
@@ -885,7 +886,7 @@ stopping hbase...............</programli
   &lt;property&gt;
     &lt;name&gt;hbase.rootdir&lt;/name&gt;
     &lt;value&gt;hdfs://example0:9000/hbase&lt;/value&gt;
-    &lt;description&gt;The directory shared by region servers.
+    &lt;description&gt;The directory shared by RegionServers.
     &lt;/description&gt;
   &lt;/property&gt;
   &lt;property&gt;
@@ -904,8 +905,8 @@ stopping hbase...............</programli
         <section xml:id="regionservers">
           <title><filename>regionservers</filename></title>
 
-          <para>In this file you list the nodes that will run regionservers.
-          In our case we run regionservers on all but the head node
+          <para>In this file you list the nodes that will run RegionServers.
+          In our case we run RegionServers on all but the head node
           <varname>example1</varname> which is carrying the HBase Master and
           the HDFS namenode</para>
 

Modified: hbase/branches/0.90/src/docbkx/performance.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.90/src/docbkx/performance.xml?rev=1099696&r1=1099695&r2=1099696&view=diff
==============================================================================
--- hbase/branches/0.90/src/docbkx/performance.xml (original)
+++ hbase/branches/0.90/src/docbkx/performance.xml Thu May  5 06:21:55 2011
@@ -15,11 +15,25 @@
   factors involved; RAM, compression, JVM settings, etc. Afterward, come back
   here for more pointers.</para>
 
+  <note xml:id="rpc.logging"><title>Enabling RPC-level logging</title>
+  <para>Enabling the RPC-level logging on a RegionServer can often given
+      insight on timings at the server.  Once enabled, the amount of log
+      spewed is voluminous.  It is not recommended that you leave this
+      logging on for more than short bursts of time.  To enable RPC-level
+      logging, browse to the RegionServer UI and click on 
+      <emphasis>Log Level</emphasis>.  Set the log level to <varname>DEBUG</varname> for the package
+      <classname>org.apache.hadoop.ipc</classname> (Thats right, for
+      hadoop.ipc, NOT, hbase.ipc).  Then tail the RegionServers log.
+      Analyze.</para>
+  <para>To disable, set the logging level back to <varname>INFO</varname> level.
+  </para>
+  </note>
+
   <section xml:id="jvm">
     <title>Java</title>
 
     <section xml:id="gc">
-      <title>The Garage Collector and HBase</title>
+      <title>The Garbage Collector and HBase</title>
 
       <section xml:id="gcpause">
         <title>Long GC pauses</title>
@@ -46,16 +60,14 @@
   <section xml:id="perf.configurations">
     <title>Configurations</title>
 
-    <para>See the section on <link
-    linkend="recommended_configurations">recommended
-    configurations</link>.</para>
+    <para>See <xref linkend="recommended_configurations" />.</para>
 
     <section xml:id="perf.number.of.regions">
       <title>Number of Regions</title>
 
-      <para>The number of regions for an HBase table is driven by the <link
-      linkend="bigger.regions">filesize</link>. Also, see the architecture
-      section on <link linkend="arch.regions.size">region size</link></para>
+      <para>The number of regions for an HBase table is driven by the <xref
+              linkend="bigger.regions" />. Also, see the architecture
+          section on <xref linkend="arch.regions.size" /></para>
     </section>
 
     <section xml:id="perf.compactions.and.splits">
@@ -68,18 +80,60 @@
 
     <section xml:id="perf.compression">
       <title>Compression</title>
-
-      <para>Production systems should use compression such as <link
-      linkend="lzo">LZO</link> compression with their column family
+      <para>Production systems should use compression such as <xref linkend="lzo" /> compression with their column family
       definitions.</para>
     </section>
+
+    <section xml:id="perf.handlers">
+        <title><varname>hbase.regionserver.handler.count</varname></title>
+        <para>See <xref linkend="hbase.regionserver.handler.count"/>. 
+            This setting in essence sets how many requests are
+            concurrently being processed inside the RegionServer at any
+            one time.  If set too high, then throughput may suffer as
+            the concurrent requests contend; if set too low, requests will
+            be stuck waiting to get into the machine.  You can get a
+            sense of whether you have too little or too many handlers by
+            <xref linkend="rpc.logging" />
+            on an individual RegionServer then tailing its logs (Queued requests
+            consume memory).</para>
+    </section>
+    <section xml:id="perf.hfile.block.cache.size">
+        <title><varname>hfile.block.cache.size</varname></title>
+        <para>See <xref linkend="hfile.block.cache.size"/>. 
+        A memory setting for the RegionServer process.
+        </para>
+    </section>    
+    <section xml:id="perf.rs.memstore.upperlimit">
+        <title><varname>hbase.regionserver.global.memstore.upperLimit</varname></title>
+        <para>See <xref linkend="hbase.regionserver.global.memstore.upperLimit"/>.  
+        This memory setting is often adjusted for the RegionServer process depending on needs.
+        </para>
+    </section>    
+    <section xml:id="perf.rs.memstore.lowerlimit">
+        <title><varname>hbase.regionserver.global.memstore.lowerLimit</varname></title>
+        <para>See <xref linkend="hbase.regionserver.global.memstore.lowerLimit"/>.  
+        This memory setting is often adjusted for the RegionServer process depending on needs.
+        </para>
+    </section>
+    <section xml:id="perf.hstore.blockingstorefiles">
+        <title><varname>hbase.hstore.blockingStoreFiles</varname></title>
+        <para>See <xref linkend="hbase.hstore.blockingStoreFiles"/>.  
+        If there is blocking in the RegionServer logs, increasing this can help.
+        </para>
+    </section>
+    <section xml:id="perf.hregion.memstore.block.multiplier">
+        <title><varname>hbase.hregion.memstore.block.multiplier</varname></title>
+        <para>See <xref linkend="hbase.hregion.memstore.block.multiplier"/>.  
+        If there is enough RAM, increasing this can help.  
+        </para>
+    </section>
+
   </section>
 
   <section xml:id="perf.number.of.cfs">
     <title>Number of Column Families</title>
 
-    <para>See the section on <link linkend="number.of.cfs">Number of Column
-    Families</link>.</para>
+    <para>See <xref linkend="number.of.cfs" />.</para>
   </section>
 
   <section xml:id="perf.one.region">
@@ -92,9 +146,47 @@
 
   <section xml:id="perf.batch.loading">
     <title>Batch Loading</title>
+    <para>Use the bulk load tool if you can.  See
+        <link xlink:href="http://hbase.apache.org/bulk-loads.html">Bulk Loads</link>.
+        Otherwise, pay attention to the below.
+    </para>
+
+  <section xml:id="precreate.regions">
+  <title>
+  Table Creation: Pre-Creating Regions
+  </title>
+<para>
+Tables in HBase are initially created with one region by default.  For bulk imports, this means that all clients will write to the same region until it is large enough to split and become distributed across the cluster.  A useful pattern to speed up the bulk import process is to pre-create empty regions.  Be somewhat conservative in this, because too-many regions can actually degrade performance.  An example of pre-creation using hex-keys is as follows (note:  this example may need to be tweaked to the individual applications keys):
+</para>
+<para>
+<programlisting>public static boolean createTable(HBaseAdmin admin, HTableDescriptor table, byte[][] splits)
+throws IOException {
+  try {
+    admin.createTable( table, splits );
+    return true;
+  } catch (TableExistsException e) {
+    logger.info("table " + table.getNameAsString() + " already exists");
+    // the table already exists...
+    return false;  
+  }
+}
 
-    <para>See the section on <link linkend="precreate.regions">Pre Creating
-    Regions</link> as well as bulk loading</para>
+public static byte[][] getHexSplits(String startKey, String endKey, int numRegions) {
+  byte[][] splits = new byte[numRegions-1][];
+  BigInteger lowestKey = new BigInteger(startKey, 16);
+  BigInteger highestKey = new BigInteger(endKey, 16);
+  BigInteger range = highestKey.subtract(lowestKey);
+  BigInteger regionIncrement = range.divide(BigInteger.valueOf(numRegions));
+  lowestKey = lowestKey.add(regionIncrement);
+  for(int i=0; i &lt; numRegions-1;i++) {
+    BigInteger key = lowestKey.add(regionIncrement.multiply(BigInteger.valueOf(i)));
+    byte[] b = String.format("%016x", key).getBytes();
+    splits[i] = b;
+  }
+  return splits;
+}</programlisting>
+  </para>
+  </section>
   </section>
 
   <section>
@@ -107,7 +199,7 @@
       to false on your <link
       xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html">HTable</link>
       instance. Otherwise, the Puts will be sent one at a time to the
-      regionserver. Puts added via <code> htable.add(Put)</code> and <code> htable.add( &lt;List&gt; Put)</code>
+      RegionServer. Puts added via <code> htable.add(Put)</code> and <code> htable.add( &lt;List&gt; Put)</code>
       wind up in the same write buffer. If <code>autoFlush = false</code>,
       these messages are not sent until the write-buffer is filled. To
       explicitly flush the messages, call <methodname>flushCommits</methodname>.
@@ -127,7 +219,17 @@
       processed. Setting this value to 500, for example, will transfer 500
       rows at a time to the client to be processed. There is a cost/benefit to
       have the cache value be large because it costs more in memory for both
-      client and regionserver, so bigger isn't always better.</para>
+      client and RegionServer, so bigger isn't always better.</para>
+    </section>
+    <section xml:id="perf.hbase.client.selection">
+      <title>Scan Attribute Selection</title>
+
+      <para>Whenever a Scan is used to process large numbers of rows (and especially when used
+      as a MapReduce source), be aware of which attributes are selected.   If <code>scan.addFamily</code> is called
+      then <emphasis>all</emphasis> of the attributes in the specified ColumnFamily will be returned to the client.
+      If only a small number of the available attributes are to be processed, then only those attributes should be specified
+      in the input scan because attribute over-selection is a non-trivial performance penalty over large datasets.
+      </para>
     </section>
 
     <section xml:id="perf.hbase.client.scannerclose">
@@ -137,7 +239,7 @@
       <emphasis>avoiding</emphasis> performance problems. If you forget to
       close <link
       xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/ResultScanner.html">ResultScanners</link>
-      you can cause problems on the regionservers. Always have ResultScanner
+      you can cause problems on the RegionServers. Always have ResultScanner
       processing enclosed in try/catch blocks... <programlisting>
 Scan scan = new Scan();
 // set attrs...
@@ -156,10 +258,21 @@ htable.close();</programlisting></para>
 
       <para><link
       xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html">Scan</link>
-      instances can be set to use the block cache in the region server via the
+      instances can be set to use the block cache in the RegionServer via the
       <methodname>setCacheBlocks</methodname> method. For input Scans to MapReduce jobs, this should be
       <varname>false</varname>. For frequently accessed rows, it is advisable to use the block
       cache.</para>
     </section>
+    <section xml:id="perf.hbase.client.rowkeyonly">
+      <title>Optimal Loading of Row Keys</title>
+      <para>When performing a table <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html">scan</link>
+            where only the row keys are needed (no families, qualifiers, values or timestamps), add a FilterList with a
+            <varname>MUST_PASS_ALL</varname> operator to the scanner using <methodname>setFilter</methodname>. The filter list
+            should include both a <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/FirstKeyOnlyFilter.html">FirstKeyOnlyFilter</link>
+            and a <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/KeyOnlyFilter.html">KeyOnlyFilter</link>.
+            Using this filter combination will result in a worst case scenario of a RegionServer reading a single value from disk
+            and minimal network traffic to the client for a single row.
+      </para>
+    </section>
   </section>
 </chapter>

Modified: hbase/branches/0.90/src/docbkx/preface.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.90/src/docbkx/preface.xml?rev=1099696&r1=1099695&r2=1099696&view=diff
==============================================================================
--- hbase/branches/0.90/src/docbkx/preface.xml (original)
+++ hbase/branches/0.90/src/docbkx/preface.xml Thu May  5 06:21:55 2011
@@ -23,4 +23,25 @@
   hope to fill in the holes with time. Feel free to add to this book by adding
   a patch to an issue up in the HBase <link
   xlink:href="https://issues.apache.org/jira/browse/HBASE">JIRA</link>.</para>
+
+  <note xml:id="headsup">
+      <title>Heads-up</title>
+      <para>
+          If this is your first foray into the wonderful world of
+          Distributed Computing, then you are in for
+          some interesting times.  First off, distributed systems are
+          hard; making a distributed system hum requires a disparate
+          skillset that needs span systems (hardware and software) and
+          networking.  Your cluster' operation can hiccup because of any
+          of a myriad set of reasons from bugs in HBase itself through misconfigurations
+          -- misconfiguration of HBase but also operating system misconfigurations --
+          through to hardware problems whether it be a bug in your network card
+          drivers or an underprovisioned RAM bus (to mention two recent
+          examples of hardware issues that manifested as "HBase is slow").
+          You will also need to do a recalibration if up to this your
+          computing has been bound to a single box.  Here is one good
+          starting point:
+          <link xlink:href="http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing">Fallacies of Distributed Computing</link>.
+      </para>
+  </note>
 </preface>

Modified: hbase/branches/0.90/src/docbkx/shell.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.90/src/docbkx/shell.xml?rev=1099696&r1=1099695&r2=1099696&view=diff
==============================================================================
--- hbase/branches/0.90/src/docbkx/shell.xml (original)
+++ hbase/branches/0.90/src/docbkx/shell.xml Thu May  5 06:21:55 2011
@@ -24,7 +24,7 @@
             arguments are entered into the
             HBase shell; in particular note how table names, rows, and
             columns, etc., must be quoted.</para>
-            <para>See <link linkend="shell_exercises">Shell Exercises</link>
+        <para>See <xref linkend="shell_exercises" />
             for example basic shell operation.</para>
 
     <section xml:id="scripting"><title>Scripting</title>

Modified: hbase/branches/0.90/src/docbkx/upgrading.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.90/src/docbkx/upgrading.xml?rev=1099696&r1=1099695&r2=1099696&view=diff
==============================================================================
--- hbase/branches/0.90/src/docbkx/upgrading.xml (original)
+++ hbase/branches/0.90/src/docbkx/upgrading.xml Thu May  5 06:21:55 2011
@@ -9,8 +9,7 @@
       xmlns:db="http://docbook.org/ns/docbook">
     <title>Upgrading</title>
     <para>
-    Review the <link linkend="requirements">requirements</link>
-    section above, in particular the section on Hadoop version.
+        Review <xref linkend="requirements" />, in particular the section on Hadoop version.
     </para>
     <section xml:id="upgrade0.90">
     <title>Upgrading to HBase 0.90.x from 0.20.x or 0.89.x</title>
@@ -30,7 +29,7 @@
              HBase jar and read from there.  If you would like to review
              the content of this file, see it in the src tree at
              <filename>src/main/resources/hbase-default.xml</filename> or
-             see <link linkend="hbase_default_configurations">Default HBase Configurations</link>.
+             see <xref linkend="hbase_default_configurations" />.
           </para>
           <para>
             Finally, if upgrading from 0.20.x, check your 



Mime
View raw message