accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mwa...@apache.org
Subject [accumulo-website] branch master updated: Updated MapReduce docs to include new configuration (#135)
Date Wed, 28 Nov 2018 19:13:45 GMT
This is an automated email from the ASF dual-hosted git repository.

mwalch pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/master by this push:
     new 2bab009  Updated MapReduce docs to include new configuration (#135)
2bab009 is described below

commit 2bab009eda7c4786a1b01f2c2e223c91072c30fb
Author: Mike Walch <mwalch@apache.org>
AuthorDate: Wed Nov 28 14:13:41 2018 -0500

    Updated MapReduce docs to include new configuration (#135)
---
 _docs-2/administration/upgrading.md |  1 +
 _docs-2/development/mapreduce.md    | 15 +++++++++++++++
 2 files changed, 16 insertions(+)

diff --git a/_docs-2/administration/upgrading.md b/_docs-2/administration/upgrading.md
index 4ad5a00..18c5714 100644
--- a/_docs-2/administration/upgrading.md
+++ b/_docs-2/administration/upgrading.md
@@ -42,6 +42,7 @@ Below are some changes in 2.0 that you should be aware of:
     - `log4j-service.properties` for all Accumulo services (except monitor)
     - `logj4-monitor.properties` for Accumulo monitor
     - `log4j.properties` for Accumulo clients and commands
+* [New Hadoop configuration is required]({% durl development/mapreduce#configuration %})
when reading or writing to Accumulo using MapReduce.
 * Run the command `accumulo shell` to access the shell using configuration in `conf/accumulo-client.properties`
 
 When your Accumulo 2.0 installation is properly configured, stop Accumulo 1.8/9 and start
Accumulo 2.0:
diff --git a/_docs-2/development/mapreduce.md b/_docs-2/development/mapreduce.md
index 0adb649..ee6a93a 100644
--- a/_docs-2/development/mapreduce.md
+++ b/_docs-2/development/mapreduce.md
@@ -13,6 +13,21 @@ can be set via these two format classes to do the following:
 * Restrict the scan to a range of rows
 * Restrict the input to a subset of available columns
 
+## Configuration
+
+Since 2.0.0, Accumulo no longer has the same versions of dependencies (i.e Guava, etc) as
Hadoop.
+When launching a MapReduce job that reads or writes to Accumulo, you should build a shaded
jar
+with all of your dependencies and complete the following steps so YARN only includes Hadoop
code
+(and not all of Hadoop dependencies) when running your MapReduce job:
+
+1. Set `export HADOOP_USE_CLIENT_CLASSLOADER=true` in your environment before submitting
+   your job with `yarn` command.
+
+2. Set the following in your Job configuration.
+    ```java
+    job.getConfiguration().set("mapreduce.job.classloader", "true");
+    ```
+
 ## Mapper and Reducer classes
 
 To read from an Accumulo table create a Mapper with the following class


Mime
View raw message