accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From build...@apache.org
Subject svn commit: r924532 - in /websites/staging/accumulo/trunk/content: ./ 1.6/accumulo_user_manual.html
Date Fri, 03 Oct 2014 16:22:57 GMT
Author: buildbot
Date: Fri Oct  3 16:22:57 2014
New Revision: 924532

Log:
Staging update by buildbot for accumulo

Modified:
    websites/staging/accumulo/trunk/content/   (props changed)
    websites/staging/accumulo/trunk/content/1.6/accumulo_user_manual.html

Propchange: websites/staging/accumulo/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Fri Oct  3 16:22:57 2014
@@ -1 +1 @@
-1627921
+1629249

Modified: websites/staging/accumulo/trunk/content/1.6/accumulo_user_manual.html
==============================================================================
--- websites/staging/accumulo/trunk/content/1.6/accumulo_user_manual.html (original)
+++ websites/staging/accumulo/trunk/content/1.6/accumulo_user_manual.html Fri Oct  3 16:22:57
2014
@@ -548,6 +548,8 @@ span.icon > [class^="icon-"], span.icon 
 <li><a href="#_cluster_specification">11.5.3. Cluster Specification</a></li>
 <li><a href="#_accumulo_settings">11.5.4. Accumulo Settings</a></li>
 <li><a href="#_deploy_configuration">11.5.5. Deploy Configuration</a></li>
+<li><a href="#_sensitive_configuration_values">11.5.6 Sensitive Configuration
Values</a></li>
+<li><a href="#_using_a_javakeystorecredentialprovider_for_storage">11.5.7. Using
a JavaKeyStoreCredentialProvider for storage</a></li>
 </ul>
 </li>
 <li><a href="#_initialization">11.6. Initialization</a></li>
@@ -597,7 +599,8 @@ span.icon > [class^="icon-"], span.icon 
 <li><a href="#zookeeper_failure">13.8.2. ZooKeeper Failure</a></li>
 </ul>
 </li>
-<li><a href="#_file_naming_conventions">13.9. File Naming Conventions</a></li>
+<li><a href="#_upgrade_issues">13.9. Upgrade Issues</a></li>
+<li><a href="#_file_naming_conventions">13.10. File Naming Conventions</a></li>
 </ul>
 </li>
 <li><a href="#configuration">14. Appendix A: Configuration Management</a></li>
@@ -1296,6 +1299,17 @@ Instance inst = new ZooKeeperInstance(in
 Connector conn = inst.getConnector("user", new PasswordToken("passwd"));</code></pre>
 </div>
 </div>
+<div class="paragraph">
+<p>The PasswordToken is the most common implementation of an <code>AuthenticationToken</code>.
+This general interface allows authentication as an Accumulo user to come from a variety of
sources
+or means. The CredentialProviderToken leverages the Hadoop CredentialProviders (new in Hadoop
2.6).</p>
+</div>
+<div class="paragraph">
+<p>For example, the CredentialProviderToken can be used in conjunction with a Java
KeyStore
+to alleviate passwords stored in cleartext. When stored in HDFS, a single KeyStore can be
+used across an entire instance. Be aware that KeyStores stored on the local filesystem
+must be made available to all nodes in the Accumulo cluster.</p>
+</div>
 </div>
 <div class="sect2">
 <h3 id="_writing_data">4.3. Writing Data</h3>
@@ -1405,10 +1419,10 @@ Scanner scan =
     conn.createScanner("table", auths);
 
 scan.setRange(new Range("harry","john"));
-scan.fetchFamily("attributes");
+scan.fetchColumnFamily(new Text("attributes"));
 
 for(Entry&lt;Key,Value&gt; entry : scan) {
-    String row = entry.getKey().getRow();
+    Text row = entry.getKey().getRow();
     Value value = entry.getValue();
 }</code></pre>
 </div>
@@ -1478,7 +1492,7 @@ TabletServers in parallel.</p>
 BatchScanner bscan =
     conn.createBatchScanner("table", auths, 10);
 bscan.setRanges(ranges);
-bscan.fetchFamily("attributes");
+bscan.fetchColumnFamily(new Text("attributes"));
 
 for(Entry&lt;Key,Value&gt; entry : scan) {
     System.out.println(entry.getValue());
@@ -1944,6 +1958,14 @@ config -t mytable -s table.iterator.minc
 config -t mytable -s table.iterator.majc.myiter.opt.myoptionname=myoptionvalue</pre>
 </div>
 </div>
+<div class="paragraph">
+<p>Typically, a table will have multiple iterators. Accumulo configures a set of
+system level iterators for each table. These iterators provide core functionality
+like visibility label filtering and may not be removed by users. User level iterators
+are applied in the order of their priority. Priority is a user-configured integer;
+iterators with lower numbers go first, passing the results of their iteration on
+to the other iterators up the stack.</p>
+</div>
 </div>
 <div class="sect3">
 <h4 id="_setting_iterators_programmatically">6.4.2. Setting Iterators Programmatically</h4>
@@ -3569,8 +3591,14 @@ mutation.put(colFam, colQual, colVis, ti
 <div class="paragraph">
 <p>Security labels consist of a set of user-defined tokens that are required to read
the
 value the label is associated with. The set of tokens required can be specified using
-syntax that supports logical AND and OR combinations of tokens, as well as nesting
-groups of tokens together.</p>
+syntax that supports logical AND <code>&</code> and OR <code>|</code>
combinations of tokens, as well as nesting
+groups <code>()</code> of tokens together.</p>
+</div>
+<div class="paragraph">
+<p>Each term is comprised of one to many alpha-numeric characters, hyphens, underscores
+or periods. Optionally, each term may be wrapped in quotation marks which removes the
+restriction on valid characters. In quoted terms, quotation marks and backslash characters
+can be used as characters in the term by escaping them with a backslash.</p>
 </div>
 <div class="paragraph">
 <p>For example, suppose within our organization we want to label our data values with
@@ -3926,17 +3954,20 @@ native map also avoids the performance i
 in the JVM by causing it to pause much less frequently.</p>
 </div>
 <div class="paragraph">
-<p>32-bit and 64-bit Linux versions of the native map ship with the Accumulo dist package.
-For other operating systems, the native map can be built from the codebase in two ways-
-from maven or from the Makefile.</p>
+<p>32-bit and 64-bit Linux and Mac OS X versions of the native map can be built from
the
+Accumulo bin package by executing <code>$ACCUMULO_HOME/bin/build_native_library.sh</code>.
+If your system's default compiler options are insufficient, you can add additional compiler
+options to the command line, such as options for the architecture. These will be passed
+to the Makefile in the environment variable USERFLAGS.</p>
+<p>Examples:</p>
 </div>
 <div class="olist arabic">
 <ol class="arabic">
 <li>
-<p>Build from maven using the following command: <code>mvn clean package -Pnative</code>.</p>
+<p><code>$ACCUMULO_HOME/bin/build_native_library.sh</code></p>
 </li>
 <li>
-<p>Build from the c++ source by running <code>make</code> in the <code>$ACCUMULO_HOME/server/src/main/c++</code>
directory.</p>
+<p><code>$ACCUMULO_HOME/bin/build_native_library.sh -m32</code></p>
 </li>
 </ol>
 </div>
@@ -3944,7 +3975,9 @@ from maven or from the Makefile.</p>
 <p>After building the native map from the source, you will find the artifact in
 <code>$ACCUMULO_HOME/lib/native</code>. Upon starting up, the tablet server will
look
 in this directory for the map library. If the file is renamed or moved from its
-target directory, the tablet server may not be able to find it.</p>
+target directory, the tablet server may not be able to find it. The system can also
+locate the native maps shared library by setting <code>LD_LIBRARY_PATH</code>
(or 
+<code>DYLD_LIBRARY_PATH</code> on Mac OS X) in <code>$ACCUMULO_HOME/conf/accumulo-env.sh</code></p>
 </div>
 </div>
 <div class="sect3">
@@ -3997,7 +4030,9 @@ settings between processes and helps fin
 </div>
 <div class="paragraph">
 <p>The instance needs a secret to enable secure communication between servers. Configure
your
-secret and make sure that the <code>accumulo-site.xml</code> file is not readable
to other users.</p>
+secret and make sure that the <code>accumulo-site.xml</code> file is not readable
to other users.
+For alternatives to storing the <code>instance.secret</code> in plaintext, please
read the
+<code>Sensitive Configuration Values</code> section.</p>
 </div>
 <div class="paragraph">
 <p>Some settings can be modified via the Accumulo shell and take effect immediately,
but
@@ -4013,6 +4048,69 @@ from the <code>$ACCUMULO_HOME/conf/</cod
 specified in the slaves file.</p>
 </div>
 </div>
+<div class="sect3">
+<h4 id="_sensitive_configuration_values">11.5.6. Sensitive Configuration Values</h4>
+<div class="paragraph">
+<p>Accumulo has a number of properties that can be specified via the accumulo-site.xml
+file which are sensitive in nature, instance.secret and trace.token.property.password
+are two common examples. Both of these properties, if compromised, have the ability
+to result in data being leaked to users who should not have access to that data.</p>
+</div>
+<div class="paragraph">
+<p>In Hadoop-2.6.0, a new CredentialProvider class was introduced which serves as a
common
+implementation to abstract away the storage and retrieval of passwords from plaintext
+storage in configuration files. Any Property marked with the <code>Sensitive</code>
annotation
+is a candidate for use with these CredentialProviders. For version of Hadoop which lack
+these classes, the feature will just be unavailable for use.</p>
+</div>
+<div class="paragraph">
+<p>A comma separated list of CredentialProviders can be configured using the Accumulo
Property
+<code>general.security.credential.provider.paths</code>. Each configured URL
will be consulted
+when the Configuration object for accumulo-site.xml is accessed.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_using_a_javakeystorecredentialprovider_for_storage">11.5.7. Using a JavaKeyStoreCredentialProvider
for storage</h4>
+<div class="paragraph">
+<p>One of the implementations provided in Hadoop-2.6.0 is a Java KeyStore CredentialProvider.
+Each entry in the KeyStore is the Accumulo Property key name. For example, to store the
+\texttt{instance.secret}, the following command can be used:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>hadoop credential create instance.secret --provider jceks://file/etc/accumulo/conf/accumulo.jceks</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The command will then prompt you to enter the secret to use and create a keystore
in:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>/etc/accumulo/conf/accumulo.jceks</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Then, accumulo-site.xml must be configured to use this KeyStore as a CredentialProvider:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="xml language-xml">&lt;property&gt;
+    &lt;name&gt;general.security.credential.provider.paths&lt;/name&gt;
+    &lt;value&gt;jceks://file/etc/accumulo/conf/accumulo.jceks&lt;/value&gt;
+&lt;/property&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>This configuration will then transparently extract the <code>instance.secret</code>
from
+the configured KeyStore and alleviates a human readable storage of the sensitive
+property.</p>
+</div>
+<div class="paragraph">
+<p>A KeyStore can also be stored in HDFS, which will make the KeyStore readily available
to
+all Accumulo servers. If the local filesystem is used, be aware that each Accumulo server
+will expect the KeyStore in the same location.</p>
+</div>
+</div>
 </div>
 <div class="sect2">
 <h3 id="_initialization">11.6. Initialization</h3>
@@ -5344,7 +5442,7 @@ the gap between how many splits the old 
 <div class="paragraph">
 <p>For each table, make a new directory that you can move (or copy if you have the
HDFS space to do so)
 all of the rfiles for a given table into. For example, to process the table with an ID of
<code>1</code>, make a new directory,
-say <code>/new-table-1</code> and then copy all files from <code>/accumulo/tables/1/<strong>/</strong>.rf</code>
into that directory. Additionally,
+say <code>/new-table-1</code> and then copy all files from <code>/accumulo/tables/1/*/*.rf</code>
into that directory. Additionally,
 make a directory, <code>/new-table-1-failures</code>, for any failures during
the import process. Then, issue the import
 command using the Accumulo shell into the new table, telling Accumulo to not re-set the timestamp:</p>
 </div>
@@ -5380,7 +5478,25 @@ omission of new data.</p>
 </div>
 </div>
 <div class="sect2">
-<h3 id="_file_naming_conventions">13.9. File Naming Conventions</h3>
+<h3 id="_upgrade_issues">13.9. Upgrade Issues</h3>
+<div class="paragraph">
+<p><strong>Q</strong>: I upgrade from 1.4 to 1.5 to 1.6 but still have
some WAL files on local disk. Do I have any way
+to recover them?</p>
+</div>
+<div class="paragraph">
+<p><strong>A</strong>: Yes, you can recover them by running the LocalWALRecovery
utility on each node that needs
+recovery performed. The utility will default to using the directory specified by <code>logger.dir.walog</code>
in your
+configuration, or can be overriden by using the <code>--local-wal-directories</code>
option on the tool. It can be
+invoked as follows:</p>
+<div class="literalblock">
+<div class="content">
+<pre>$ACCUMULO_HOME/bin/accumulo org.apache.accumulo.tserver.log.LocalWALRecovery</pre>
+</div>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_file_naming_conventions">13.10. File Naming Conventions</h3>
 <div class="paragraph">
 <p><strong>Q</strong>: Why are files named like they are? Why do some start
with <code>C</code> and others with <code>F</code>?</p>
 </div>



Mime
View raw message