accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject svn commit: r1591779 - in /accumulo/site/branches/redesign14: ./ content/bylaws.mdtext content/glossary.mdtext content/release_notes/1.6.0.mdtext
Date Thu, 01 May 2014 21:02:38 GMT
Author: bhavanki
Date: Thu May  1 21:02:38 2014
New Revision: 1591779

Merged trunk into redesign14 branch - bylaw update, glossary update, relnotes 1.6.0 update.

    accumulo/site/branches/redesign14/   (props changed)

Propchange: accumulo/site/branches/redesign14/
  Merged /accumulo/site/trunk:r1587699-1591778

Modified: accumulo/site/branches/redesign14/content/bylaws.mdtext
--- accumulo/site/branches/redesign14/content/bylaws.mdtext (original)
+++ accumulo/site/branches/redesign14/content/bylaws.mdtext Thu May  1 21:02:38 2014
@@ -17,7 +17,7 @@ Notice:    Licensed to the Apache Softwa
            specific language governing permissions and limitations
            under the License.
-This is version 2 of the bylaws. Community work actively continues on the bylaws, and so
key segments of them are subject to change.
+This is version 3 of the bylaws. Community work actively continues on the bylaws, and so
key segments of them are subject to change.
 # Introduction
@@ -210,7 +210,7 @@ For Code Change actions, a committer may
 <tr><td>Modifying Bylaws</td>
     <td>Modifying this document.</td>
-    <td>Majority approval</td>
+    <td>Consensus approval</td>
     <td>Active PMC members</td>

Modified: accumulo/site/branches/redesign14/content/glossary.mdtext
--- accumulo/site/branches/redesign14/content/glossary.mdtext (original)
+++ accumulo/site/branches/redesign14/content/glossary.mdtext Thu May  1 21:02:38 2014
@@ -37,7 +37,7 @@ Notice:    Licensed to the Apache Softwa
 <dt>iterator scopes</dt>
 <dd>the possible scopes for iterators are where the tablet server is already reading
and/or writing data: minor compaction / flush time (<em>minc</em> scope), major
compaction / file merging time (<em>majc</em> scope), and query time (<em>scan</em>
+<dd>process that identifies temporary files in HDFS that are no longer needed by any
process, and deletes them.</dd>
 <dd>the key into the distributed sorted map which is accumulo.  The key is subdivided
into row, column, and timestamp.  The column is further divided into  family, qualifier, and
 <dt>locality group</dt>
@@ -45,19 +45,19 @@ Notice:    Licensed to the Apache Softwa
 <dt>log-structured merge-tree</dt>
 <dd>the sorting / flushing / merging scheme on which BigTable's design is based.</dd>
+<dd>in 1.4 and older, process that accepts updates to tablet servers and writes them
to local on-disk storage for redundancy. in 1.5 the functionality was subsumed by the tablet
server and datanode with HDFS writes.</dd>
 <dt>major compaction</dt>
 <dd>merging multiple files into a single file.  If all of a tablet's files are merged
into a single file, it is called a <em>full major compaction</em>.</dd>
+<dd>process that detects and responds to tablet failures, balances load across tablet
servers by assigning and migrating tablets when required, coordinates table operations, and
handles tablet server logistics (startup, shutdown, recovery).</dd>
 <dt>minor compaction</dt>
 <dd>flushing data from memory to disk.  Usually this creates a new file for a tablet,
but if the memory flushed is merge-sorted in with data from an existing file (replacing that
file), it is called a <em>merging minor compaction</em>.</dd>
+<dd>process that displays status and usage information for all Accumulo components.</dd>
 <dd>administrative abilities that must be given to a user such as creating tables or
users and changing permissions or configuration parameters.</dd>
-<dd>the portion of the key that is controls atomicity.  Keys with the same row are
guaranteed to remain on a single tablet hosted by a single tablet server, therefore multiple
key/value pairs can be added to or removed from a row at the same time. The row is used for
the primary sorting of the key.</dd>
+<dd>the portion of the key that controls atomicity.  Keys with the same row are guaranteed
to remain on a single tablet hosted by a single tablet server, therefore multiple key/value
pairs can be added to or removed from a row at the same time. The row is used for the primary
sorting of the key.</dd>
 <dd>reading a range of key/value pairs.</dd>
@@ -68,4 +68,5 @@ Notice:    Licensed to the Apache Softwa
 <dd>the portion of the key that controls versioning.  Otherwise identical keys with
differing timestamps are considered to be versions of a single <em>cell</em>.
 Accumulo can be configured to keep the <em>N</em> newest versions of each <em>cell</em>.
 When a deletion entry is inserted, it deletes all earlier versions for its cell.</dd>
 <dd>immutable bytes associated with a particular key.</dd>
\ No newline at end of file

Modified: accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext
--- accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext (original)
+++ accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext Thu May  1 21:02:38
@@ -1,4 +1,4 @@
+Title: Release Notes for Apache Accumulo 1.6.0
 Notice:    Licensed to the Apache Software Foundation (ASF) under one
            or more contributor license agreements.  See the NOTICE file
            distributed with this work for additional information
@@ -24,7 +24,7 @@ Accumulo 1.6.0 runs on Hadoop 1, however
 ## Notable Improvements
-### Multiple namenode support
+### Multiple volume support
 [BigTable's][1] design allows for its internal metadata to automatically spread across multiple
nodes.  Accumulo has followed this design and scales very well as a result.  There is one
impediment to scaling though, and this is the HDFS namenode.  There are two problems with
the namenode when it comes to scaling.  First, the namenode stores all of its filesystem metadata
in memory on a single machine.  This introduces an upper bound on the number of files Accumulo
can have.  Second, there is an upper bound on the number of file operations per second that
a single namenode can support.  For example, a namenode can only support a few thousand delete
or create file request per second.  
@@ -46,7 +46,9 @@ The only built in test that conditional 
 ### Encryption
-Support for encrypting Accumulo's persistent and over the wire data was added.   [ACCUMULO-998][ACCUMULO-998],
[ACCUMULO-958][ACCUMULO-958], and [ACCUMULO-980][ACCUMULO-980] cover encrypting data at rest
in write ahead logs and rfiles.   [ACCUMULO-1009][ACCUMULO-1009] covers encrypting data over
the wire using SSL.  
+Encryption is still an experimental feature, but much progress has been made since 1.5.0.
 Support for encrypting rfiles and write ahead logs were added in [ACCUMULO-958][ACCUMULO-958]
and [ACCUMULO-980][ACCUMULO-980].  Support for encrypting data over the wire using SSL was
added in [ACCUMULO-1009][ACCUMULO-1009].
+When a tablet server fails, its write ahead logs are sorted and stored in HDFS.  In 1.6.0,
encrypting these sorted write ahead logs is not supported.  [ACCUMULO-981][ACCUMULO-981] addresses
this issue.  
 ### Pluggable compaction strategies
@@ -82,6 +84,42 @@ While Hadoop [does not support IPv6 netw
 Multiple bug-fixes were made to support running Accumulo over multiple HDFS instances using
ViewFS. [ACCUMULO-2047][ACCUMULO-2047] is the parent
 ticket that contains numerous fixes to enable this support.
+### Maven Plugin
+This version of Accumulo is accompanied by a new maven plugin for testing client apps ([ACCUMULO-1030][ACCUMULO-1030]).
You can execute the accumulo-maven-plugin inside your project by adding the following to your
pom.xml's build plugins section:
+      <plugin>
+        <groupId>org.apache.accumulo</groupId>
+        <artifactId>accumulo-maven-plugin</artifactId>
+        <version>1.6.0</version>
+        <configuration>
+          <instanceName>plugin-it-instance</instanceName>
+          <rootPassword>ITSecret</rootPassword>
+        </configuration>
+        <executions>
+          <execution>
+            <id>run-plugin</id>
+            <goals>
+              <goal>start</goal>
+              <goal>stop</goal>
+            </goals>
+          </execution>
+        </executions>
+      </plugin>
+This plugin is designed to work in conjunction with the maven-failsafe-plugin. A small test
instance of Accumulo will run during the pre-integration-test phase of the Maven build lifecycle,
and will be stopped in the post-integration-test phase. Your integration tests, executed by
maven-failsafe-plugin can access this instance with a MiniAccumuloInstance connector (the
plugin uses MiniAccumuloInstance, internally), as in the following example:
+      private static Connector conn;
+      @BeforeClass
+      public static void setUp() throws Exception {
+        String instanceName = "plugin-it-instance";
+        Instance instance = new MiniAccumuloInstance(instanceName, new File("target/accumulo-maven-plugin/"
+ instanceName));
+        conn = instance.getConnector("root", new PasswordToken("ITSecret"));
+      }
+This plugin is quite limited, currently only supporting an instance name and a root user
password as configuration parameters. Improvements are expected in future releases, so feedback
is welcome and appreciated (file bugs/requests under the "maven-plugin" component in the Accumulo
 ### Packaging
 One notable change that was made to the binary tarball is the purposeful omission of a pre-built
copy of the Accumulo "native map" library.
@@ -94,6 +132,12 @@ It is recommended that users invoke the 
 Be aware that you will need a C++ compiler/toolchain installed to build this library. Check
your GNU/Linux distribution documentation for the package manager command.
+#### Size-Based Constraint on New Tables
+A Constraint is an interface that can determine if a Mutation should be applied or rejected
server-side. New tables that are created in 1.6.0 will automatically have the `DefaultKeySizeConstraint`
+As performance can suffer when large Keys are inserted into a table, this Constraint will
reject any Key that is larger than 1MB. If this constraint is undesired, it can be removed
using the `constraint` shell
+command. See the help message on the command for more information.
 ### Other notable changes
  * [ACCUMULO-842][ACCUMULO-842] Added FATE administration to shell
@@ -132,6 +176,8 @@ Be aware that you will need a C++ compil
 ## Known Issues
+### Slower writes than previous Accumulo versions
 When using Accumulo 1.6 and Hadoop 2, Accumulo will call hsync() on HDFS.
 Calling hsync improves durability by ensuring data is on disk (where other older 
 Hadoop versions might lose data in the face of power failure); however, calling
@@ -145,8 +191,20 @@ mutation queues.
 For more information, see [ACCUMULO-1950][ACCUMULO-1950] and [this comment][ACCUMULO-1905-comment].
+Another possible cause of slower writes is the change in write ahead log replication 
+between 1.4 and 1.5.  Accumulo 1.4. defaulted to two loggers servers.  Accumulo 1.5 and 1.6
+write ahead logs in HDFS and default to using three datanodes.  
+### BatchWriter hold time error
+If a `BatchWriter` fails with `MutationsRejectedException` and the  message contains
+`"# server errors 1"` then it may be [ACCUMULO-2388][ACCUMULO-2388].  To confirm this look
in the tablet server logs 
+for `org.apache.accumulo.tserver.HoldTimeoutException` around the time the `BatchWriter`
+If this is happening often a possible work around is to set `general.rpc.timeout` to `240s`.
 ### Other known issues
+ * [ACCUMULO-981][ACCUMULO-981] Sorted write ahead logs are not encrypted.
  * [ACCUMULO-1507][ACCUMULO-1507] Dynamic Classloader still can't keep proper track of jars
  * [ACCUMULO-1588][ACCUMULO-1588] Monitor XML and JSON differ
  * [ACCUMULO-1628][ACCUMULO-1628] NPE on deep copied dumped memory iterator
@@ -214,13 +272,14 @@ and, in HDFS High-Availability instances
 [ACCUMULO-802]: "Table namespaces"
 [ACCUMULO-842]: "Add FATE administration
to shell"
 [ACCUMULO-958]: "Support pluggable encryption
in walogs"
-[ACCUMULO-998]: "Support encryption at
 [ACCUMULO-980]: "Support pluggable codecs
for RFile"
+[ACCUMULO-981]: "support pluggable encryption
when recovering write-ahead logs"
 [ACCUMULO-1000]: "Conditional Mutations"
 [ACCUMULO-1009]: "Support encryption
over the wire"
 [ACCUMULO-1018]: "Client does not give
informative message when user can not read table"
+[ACCUMULO-1030]: "Create a Maven plugin
to run MiniAccumuloCluster for integration testing"
 [ACCUMULO-1218]: "document the recovery
from a failed zookeeper"
 [ACCUMULO-1336]: "Add lexicoders from
Typo to Accumulo"
 [ACCUMULO-1375]: "Update README files
in proxy module."
@@ -270,6 +329,7 @@ and, in HDFS High-Availability instances
 [ACCUMULO-2262]: "Include
in process startup"
 [ACCUMULO-2334]: "Lacking fallback when
 [ACCUMULO-2360]: "Need a way to configure
TNonblockingServer.maxReadBufferBytes to prevent OOMs"
 [ACCUMULO-2408]: "metadata table not
assigned after root table is loaded"
 [ACCUMULO-2441]: "Document internal state
stored in RFile names"
 [ACCUMULO-2495]: "OOM exception didn't
bring down tserver"

View raw message