accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bhava...@apache.org
Subject [04/10] git commit: ACCUMULO-2379 Document running functional tests under MR as a different user
Date Wed, 19 Feb 2014 16:12:07 GMT
ACCUMULO-2379 Document running functional tests under MR as a different user

Functional tests might not run under MapReduce in clusters where a different user from
the Accumulo user runs mappers. Needed configuration changes are now documented in
the functional test README.


Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/ac18b569
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/ac18b569
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/ac18b569

Branch: refs/heads/master
Commit: ac18b569738b6dd8f950ee5d747af5176bbc8d13
Parents: 3898317
Author: Bill Havanki <bhavanki@cloudera.com>
Authored: Wed Feb 19 11:05:12 2014 -0500
Committer: Bill Havanki <bhavanki@cloudera.com>
Committed: Wed Feb 19 11:05:12 2014 -0500

----------------------------------------------------------------------
 test/system/auto/README | 94 ++++++++++++++++++++++++++++++--------------
 1 file changed, 65 insertions(+), 29 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/accumulo/blob/ac18b569/test/system/auto/README
----------------------------------------------------------------------
diff --git a/test/system/auto/README b/test/system/auto/README
index 4773393..45ed158 100644
--- a/test/system/auto/README
+++ b/test/system/auto/README
@@ -1,58 +1,94 @@
 Apache Accumulo Functional Tests
 
-These scripts run a series of tests against a small local accumulo instance.  To run these
scripts, 
-you must have hadoop and zookeeper installed and running.  You will need a functioning C
compiler to build 
-a shared library needed for one of the tests.  The test suite is known to run on Linux RedHat
Enterprise 
+These scripts run a series of tests against a small local accumulo instance.
+To run these scripts, you must have Hadoop and Zookeeper installed and running.
+You will need a functioning C compiler to build a shared library needed for
+one of the tests.  The test suite is known to run on Linux RedHat Enterprise
 version 5, and Mac OS X 10.5.
 
-The tests are shown as being run from the ACCUMULO_HOME directory, but they should run from
any directory.
-Make sure to create "logs" and "walogs" directories in ACCUMULO_HOME.  Also, ensure that
accumulo-env.sh
-specifies its ACCUMULO_LOG_DIR in the following way:
+How to Run
+
+The tests are shown as being run from the ACCUMULO_HOME directory, but they
+should run from any directory. Make sure to create "logs" and "walogs"
+directories in ACCUMULO_HOME.  Also, ensure that accumulo-env.sh specifies its
+ACCUMULO_LOG_DIR in the following way:
+
 test -z "$ACCUMULO_LOG_DIR"      && export ACCUMULO_LOG_DIR=$ACCUMULO_HOME/logs
 
  $ ./test/system/auto/run.py -l
- 
-Will list all the test names.  You can run the suite like this:
- 
+
+will list all the test names.  You can run the suite like this:
+
  $ ./test/system/auto/run.py
- 
+
 You can select tests using a case-insensitive regular expression:
- 
+
  $ ./test/system/auto/run.py -t simple
  $ ./test/system/auto/run.py -t SunnyDay
- 
+
 To run tests repeatedly:
 
  $ ./test/system/auto/run.py -r 3
 
-If you are attempting to debug what is causing a test to fail, you can run the tests in "verbose"
mode:
+If you are attempting to debug what is causing a test to fail, you can run the
+tests in "verbose" mode:
 
  $ python test/system/auto/run.py -t SunnyDay -v 10
- 
-If a test is failing, and you would like to examine logs from the run, you can run the test
in "dirty" 
-mode which will keep the test from cleaning up all the logs at the end of the run:
- 
+
+If a test is failing, and you would like to examine logs from the run, you can
+run the test in "dirty" mode which will keep the test from cleaning up all the
+logs at the end of the run:
+
  $ ./test/system/auto/run.py -t some.failing.test -d
- 
-If the test suite hangs, and you would like to re-run the tests starting with the last test
that failed:
- 
+
+If the test suite hangs, and you would like to re-run the tests starting with
+the last test that failed:
+
  $ ./test/system/auto/run.py -s start.over.test
- 
-If tests tend to time out (on slower hardware, for example), you can scale up the timeout
values by a
-multiplier. This example triples timeouts:
+
+If tests tend to time out (on slower hardware, for example), you can scale up
+the timeout values by a multiplier. This example triples timeouts:
 
  $ ./test/system/auto/run.py -f 3
 
-Test results are normally printed to the console, but you can send them to XML files compatible
with
-Jenkins:
+Test results are normally printed to the console, but you can send them to XML
+files compatible with Jenkins:
 
  $ ./test/system/auto/run.py -x
 
-The full test suite can take nearly an hour.  If you have a larger hadoop cluster at your
disposal, you can 
-run the tests as a map-reduce job:
+Running under MapReduce
+
+The full test suite can take nearly an hour.  If you have a larger Hadoop
+cluster at your disposal, you can run the tests as a MapReduce job:
 
  $ python test/system/auto/run.py -l > tests
  $ hadoop fs -put tests /user/hadoop/tests
- $ ./bin/accumulo org.apache.accumulo.server.test.functional.RunTests /user/hadoop/tests
/user/hadoop/results
+ $ ./bin/accumulo org.apache.accumulo.server.test.functional.RunTests \
+   /user/hadoop/tests /user/hadoop/results
+
+The example above runs every test. You can trim the tests file to include
+only the tests you wish to run.
+
+In some clusters, the user under which MR jobs run is different from the user
+under which Accumulo is installed, and this can cause failures running the
+tests. Various configuration and permission changes can be made to help the
+tests run, including the following:
+
+* Opening up directory and file permissions on each cluster node so that the MR
+  user has the same read/write privileges as the Accumulo user. Adding the MR
+  user to a shared group is one easy way to accomplish this. Access is required
+  to the Accumulo installation, log, write-ahead log, and configuration
+  directories.
+* Creating a user directory in HDFS, named after and owned by the MR user,
+  e.g., /user/mruser.
+* Setting the ZOOKEEPER_HOME and HADOOP_CONF_DIR environment variables for the
+  MR user. These can be set using the mapred.child.env property in
+  mapred-site.xml, e.g.:
+
+  <property>
+    <name>mapred.child.env</name>
+    <value>ZOOKEEPER_HOME=/path/to/zookeeper,HADOOP_CONF_DIR=/path/to/hadoop/conf</value>
+  </property>
 
- 
+Each functional test is run by a mapper, and so you can check the mapper logs
+to see any error messages tests produce.


Mime
View raw message