Reply-To: dev@accumulo.apache.org Delivered-To: mailing list commits@accumulo.apache.org Received: (qmail 55293 invoked by uid 99); 26 Oct 2016 19:37:30 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Oct 2016 19:37:30 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 6530DF16D4; Wed, 26 Oct 2016 19:37:30 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: ctubbsii@apache.org To: commits@accumulo.apache.org Date: Wed, 26 Oct 2016 19:37:55 -0000 Message-Id: <6ebc765eeb84431ea35ca76baf240faf@git.apache.org> In-Reply-To: <18acd6ed33d2446eabeec4938b66fada@git.apache.org> References: <18acd6ed33d2446eabeec4938b66fada@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [27/31] accumulo git commit: Jekyll build from gh-pages:899f3b1 archived-at: Wed, 26 Oct 2016 19:37:36 -0000 http://git-wip-us.apache.org/repos/asf/accumulo/blob/56754f27/1.8/examples/maxmutation.html ---------------------------------------------------------------------- diff --git a/1.8/examples/maxmutation.html b/1.8/examples/maxmutation.html index 46fa95c..33c588d 100644 --- a/1.8/examples/maxmutation.html +++ b/1.8/examples/maxmutation.html @@ -21,17 +21,16 @@ - - + Apache Accumulo MaxMutation Constraints Example - + - - + + + - - + + + - - + + + - - + + + - - + + + - - + + + - - + + + - - + + + - - + + + - - + + + + + + + + + + + + +

+ + +

+ +

The Accumulo ClassLoader

+ +

Dave Marion - 03 May 2014

+ +

Originally posted at https://blogs.apache.org/accumulo/

+ +

The Accumulo classloader is an integral part of the software. The classloader is created before each of the services (master, tserver, gc, etc) are started and it is set as the classloader for that service. The classloader was rewritten in version 1.5 and this article will explain the new behavior.

+ +

First, some history

+ +

The classloader in version 1.4 used a simple hierarchy of two classloaders that would load classes from locations specified by two properties. The locations specified by the “general.classpaths” property would be used to create a parent classloader and locations specified by the “general.dynamic.classpaths” property were used to create a child classloader. The child classloader would monitor the specified locations for changes and when a change occurred the child classloader would be replaced with a new instance. Classes that referenced the orphaned child classloader would continue to work and the classloader would be garbage collected when no longer referenced. The diagram below shows the relationship between the classloaders in Accumulo 1.4.

+ +

The only place where the dynamic classloader would come into play is for user iterators and their dependencies. The general advice for using this classloader would be to put the jars containing your iterators in the dynamic location. Everything else that does not change very often or would require a restart should be put into the non-dynamic location.

+ +

There are a couple of things to note about the classloader in 1.4. First, if you modified the dynamic locations too often, you would run out of perm-gen space. This is likely due to unreferenced classes not being unloaded from the JVM. This is captured in ACCUMULO-599. Secondly, when you modified files in dynamic locations within the same cycle, it would on occasion miss the second change.

+ +

Out with the old, in with the new

+ +

The Accumulo classloader was rewritten in version 1.5. It maintains the same dynamic capability and includes a couple of new features. The classloader uses Commons VFS so that it can load jars and classes from a variety of sources, including HDFS. Being able to load jars from one location (hdfs, http, etc) will make it easier to deploy changes to your cluster. Additionally, we introduced the notion of classloader contexts into Accumulo. This is not a new concept for anyone that has used an application server, but the implementation is a little different for Accumulo.

+ +

The hierarchy set up by the new classloader uses the same property names as the old classloader. In the most basic configuration the locations specified by “general.classpaths” are used to create the root of the application classloader hierarchy. This classloader is a URLClassLoader and it does not support dynamic reloading. If you only specify this property, then you are loading all of your jars from the local file system and they will not be monitored for changes. We will call this top level application classloader the SYSTEM classloader. Next, a classloader is created that supports VFS sources and reloading. The parent of this classloader is the SYSTEM classloader and we will call this the VFS classloader. If the “general.vfs.classpaths” property is set, the VFS classloader will use this location. If the property is not set, it will use the value of “general.dynamic.classpaths” with a default value of $ACCUMULO_HOME/lib/ext to support backwards compatibility. The diagram below shows the relationship between the classloaders in Accumulo 1.5.

+ +

Running Accumulo From HDFS

+ +

If you have defined “general.vfs.classpaths” in your Accumulo configuration, then you can use the bootstrap_hdfs.sh script in the bin directory to seed HDFS with the Accumulo jars. A couple of jars will remain on the local file system for starting services. Now when you start up Accumulo the master, gc, tracer, and all of the tablet servers will get their jars and classes from HDFS. The bootstrap_hdfs.sh script sets the replication on the directory, but you may want to set it higher after bootstrapping. An example configuration setting would be:

+ +

<property>
+  <name>general.vfs.classpaths</name>
+  <value>hdfs://localhost:8020/accumulo/system-classpath</value>
+  <description>Configuration for a system level vfs classloader. Accumulo jars can be configured here and loaded out of HDFS.</description>
+</property>
+

+ +

About Contexts

+ +

You can also define classloader contexts in your accumulo-site.xml file. A context is defined by a user supplied name and it references locations like the other classloader properties. When a context is defined in the configuration, it can then be applied to one or more tables. When a context is applied to a table, then a classloader is created for that context. If multiple tables use the same context, then they share the context classloader. The context classloader is a child to the VFS classloader created above.

+ +

The goal here is to enable multiple tenants to share the same Accumulo instance. For example, we may have a context called ‘app1’ which references the jars for application A. We may also have another context called app2 which references the jars for application B. By default the context classloader delegates to the VFS classloader. This behavior may be overridden as seen in the app2 example below. The context classloader also supports reloading like the VFS classloader.

+ +

<property>
+  <name>general.vfs.context.classpath.app1</name>
+  <value>hdfs://localhost:8020/applicationA/classpath/.*.jar,file:///opt/applicationA/lib/.*.jar</value>
+  <description>Application A classpath, loads jars from HDFS and local file system</description>
+</property>
+
+<property>
+  <name>general.vfs.context.classpath.app2.delegation=post</name>
+  <value>hdfs://localhost:8020/applicationB/classpath/.*.jar,http://my-webserver/applicationB/.*.jar</value>
+  <description>Application B classpath, loads jars from HDFS and HTTP, does not delegate to parent first</description>
+</property>
+

+ +

Context classloaders do not have to be defined in the accumulo-site.xml file. The “general.vfs.context.classpath.{context}” property can be defined on the table either programatically or manually in the shell. Then set the “table.classpath.context” property on your table.

+ +

Known Issues

+ +

Remember the two issues I mentioned above? Well, they are still a problem.

+ +

ACCUMULO-1507 is tracking VFS-487 for frequent modifications to files.
If you start running out of perm-gen space, take a look at ACCUMULO-599 and try applying the JVM settings for class unloading.
Additionally, there is an issue with the bootstrap_hdfs.sh script detailed in ACCUMULO-2761. There is a workaround listed in the issue.

+ +

Please email the dev list for comments and questions.

+ + + +

View all posts in the news archive

+ +

+ + + + + +

+ + http://git-wip-us.apache.org/repos/asf/accumulo/blob/56754f27/blog/2014/05/27/getting-started-with-accumulo-1.6.0.html ---------------------------------------------------------------------- diff --git a/blog/2014/05/27/getting-started-with-accumulo-1.6.0.html b/blog/2014/05/27/getting-started-with-accumulo-1.6.0.html new file mode 100644 index 0000000..4b53a35 --- /dev/null +++ b/blog/2014/05/27/getting-started-with-accumulo-1.6.0.html @@ -0,0 +1,255 @@ + + + + + + + + + + + + +Getting Started with Apache Accumulo 1.6.0 + + + + + + + + + + + + +

+ + +

+ +

Getting Started with Apache Accumulo 1.6.0

+ +

Josh Elser - 27 May 2014

+ +

Originally posted at https://blogs.apache.org/accumulo/

+ +

On May 12th, 2014, the Apache Accumulo project happily announced version 1.6.0 to the community. This is a new major release for the project which contains many numerous new features and fixes. For the full list of notable changes, I’d recommend that you check out the release notes that were published alongside the release itself. For this post, I’d like to cover some of the changes that have been made at the installation level that are a change for users who are already familiar with the project.

+ +

Download the release

+ +

Like always, you can find out releases on the our downloads page. You have the choice of downloading the source and building it yourself, or choosing the binary tarball which already contains pre-built jars for use.

+ +

Native Maps

+ +

One of the major components of the original BigTable design was an “In-Memory Map” which provided fast insert and read operations. Accumulo implements this using a C++ sorted map with a custom allocator which is invoked by the TabletServer using JNI. Each TabletServer uses its own “native” map. It is highly desirable to use this native map as it comes with a notable performance increase over a Java map (which is the fallback when the Accumulo shared library is not found) in addition to greatly reducing the TabletServer’s JVM garbage collector stress when ingesting data.

+ +

In previous versions, the binary tarball contained a pre-compiled version of the native library (under lib/native/). Shipping a compiled binary was a convenience but also left much confusion when it didn’t work on systems which had different, incompatible versions of GCC toolchains installed than what the binary was built against. As such, we have stopped bundling the pre-built shared library in favor of users building this library on their own, and instead include an accumulo-native.tar.gz file within the lib directory which contains the necessary files to build the library yourself.

+ +

To reduce the burden on users, we’ve also introduced a new script inside of the bin directory:

+ +

build_native_map.sh
+

+ +

Invoking this script will automatically unpack, build and install the native map in $ACCUMULO_HOME/lib/native. If you’ve used older versions of Accumulo, you will also notice that the library name is different in an attempt to better follow standard conventions: libaccumulo.so on Linux and libaccumulo.dylib on Mac OS X.

+ +

Example Configurations

+ +

Apache Accumulo still bundles a set of example configuration files in conf/examples. Each sub-directory contains the complete set of files to run on a single node with the named memory limitations. For example, the files contained in conf/examples/3GB/native-standalone will run Accumulo on a single node, with native maps (don’t forget to build them first!), within a total memory footprint of 3GB. Copy the contents of one of these directories into conf/ and make sure that your relevant installation details (e.g. HADOOP_PREFIX, JAVA_HOME, etc) are properly set in accumulo-env.sh. For example:

+ +

cp $ACCUMULO_HOME/conf/examples/3G/native-standalone/* $ACCUMULO_HOME/conf
+

+ +

Alternatively, a new script, bootstrap_config.sh, was also introduced that can be invoked instead of manually copying files. It will step through a few choices (memory usage, in-memory map type, and Hadoop major version), and then automatically create the configuration files for you.

+ +

$ACCUMULO_HOME/bin/bootstrap_config.sh
+

+ +

One notable change in these scripts over previous versions is that they default to using Apache Hadoop 2 packaging details, such as the Hadoop conf directory and jar locations. It is highly recommended by the community that you use Apache Accumulo 1.6.0 with at least Apache Hadoop 2.2.0, most notably, to ensure that you will not lose data in the face of power failure. If you are still running on a Hadoop 1 release (1.2.1), you will need to edit both accumulo-env.sh and accumulo-site.xml. There are comments in each file which instruct you what needs to be changed.

+ +

Starting Accumulo

+ +

Initializing and starting Accumulo hasn’t changed at all! After you have created the configuration files and, if you’re using them, built the native maps, run:

+ +

accumulo init
+

+ +

This will prompt you to name your Accumulo instance and set the Accumulo root user’s password, then start Accumulo using

+ +

$ACCUMULO_HOME/bin/start-all.sh
+

+ + + +

View all posts in the news archive

+ +

+ + + + + +

+ +