hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Date Wed, 11 Jan 2012 02:11:41 GMT

I've been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but have run into a
couple of issues. Perhaps I'm doing something wrong.

What I've done:

  - Checked out Hadoop branch-0.23 and installed artifacts into the local maven cache:

     $ cd /path/to/hadoop
     $ mvn clean install -DskipTests

  - Checked out HBase branch-0.92 and attempted to test under the maven profile for 0.23:
     $ cd /path/to/hbase
     $ mvn -Dhadoop.profile=23 clean test

The first issue I encountered was that the MiniMRCluster would not initialize:

2012-01-10 18:01:02,489 INFO  [main] mapreduce.TestTableMapReduce(153): Started mrtest
2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying ClientProtocolProvider
: org.apache.hadoop.mapred.LocalClientProtocolProvider
2012-01-10 18:01:02,494 INFO  [main] mapreduce.Cluster(116): Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider
due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner
: "test"
2012-01-10 18:01:02,494 ERROR [main] security.UserGroupInformation(1160): PriviledgedActionException
as:apurtell (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check
your configuration for mapreduce.framework.name and the correspond server addresses.
2012-01-10 18:01:02,497 INFO  [main]
 hbase.HBaseTestingUtility(1173): Stopping mini mapreduce cluster...

I filed MAPREDUCE-3654 (https://issues.apache.org/jira/browse/MAPREDUCE-3654). There could
be more going on here, but the patch on that issue let me move forward.

Now I am seeing this:

2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked org.apache.hadoop.mapred.LocalClientProtocolProvider
as the ClientProtocolProvider
2012-01-10 17:51:35,243 INFO  [main] mapreduce.TableOutputFormat(197): Created table instance
for mrtest
2012-01-10 17:51:35,249 DEBUG [main] mapreduce.JobSubmitter(347): Configuring job job_local_0001
with file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001 as
the submit dir
2012-01-10 17:51:35,251 DEBUG [main] mapreduce.JobSubmitter(565): adding the following namenodes'
delegation tokens:[hdfs://localhost.localdomain:54525]
2012-01-10 17:51:35,251 WARN  [main]
 mapreduce.JobSubmitter(139): Use GenericOptionsParser for parsing the arguments. Applications
should implement Tool for the same.
2012-01-10 17:51:35,252 DEBUG [main] mapreduce.JobSubmitter(158): default FileSystem: file:///
2012-01-10 17:51:35,261 WARN  [main] mapreduce.JobSubmitter(241): No job jar file set. 
User classes may not be found. See Job or Job#setJar(String).
2012-01-10 17:51:35,266 INFO  [main] mapreduce.JobSubmitter(388): Cleaning up the staging
area file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
2012-01-10 17:51:35,268 ERROR [main] security.UserGroupInformation(1160): PriviledgedActionException
as:apurtell (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
2012-01-10 17:51:35,273 INFO  [main] hbase.HBaseTestingUtility(1173): Stopping mini mapreduce

Of course that FileNotFoundException is for a file that does indeed exist, in the local maven

    $ ls -l /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
    -rw-rw-r-- 1 apurtell apurtell 764555 2012-01-05 18:56 /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar

I set permissions on all directories from /home/apurtell to the jar to 'rwxr-xr-x' just to
make sure there was not some kind of odd permissions problem.

Any idea what is going on here?

Best regards,

- Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
View raw message