hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahadev Konar <maha...@hortonworks.com>
Subject Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Date Fri, 13 Jan 2012 03:53:18 GMT
Andrew,
 I just checked out 0.92 branch and tried running:

mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce

Looks like a compilation issue:

------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR] /Users/mahadev/workspace/hbase-workspace/hbase-git/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java:[341,33]
cannot find symbol
[ERROR] symbol  : variable dnRegistration
[ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataNode
[ERROR] -> [Help 1]
[ERROR]
--------------

Looks like this commit caused the issue:

=====================
commit 26eb39b424838717cf878332db8c837b02117c5b
    HBASE-5163  TestLogRolling#testLogRollOnDatanodeDeath fails
sometimes on Jenkins or hadoop QA
                   ("The directory is already locked.") (N Keywal)
===================

Reverting that patch locally, we have a fix to get the tests working
(thanks to Hitesh). Mostly changes to pom files. Should we open a
HBase jira for the compilation issue? I suppose 0.92 wont be the
target fix or would it?

thanks
mahadev

On Thu, Jan 12, 2012 at 6:02 PM, Mahadev Konar <mahadev@hortonworks.com> wrote:
> HI Andrew,
>  Sorry got side tracked with something else. Taking a look now.
>
> mahadev
>
> On Thu, Jan 12, 2012 at 3:23 PM, Andrew Purtell <apurtell@apache.org> wrote:
>> Hi Mahadev,
>>
>> Was this reproducible?
>>
>> Best regards,
>>
>>   - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>> ________________________________
>> From: Andrew Purtell <apurtell@apache.org>
>> To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
>> Cc: Mahadev Konar <mahadev@hortonworks.com>
>> Sent: Wednesday, January 11, 2012 9:52 AM
>>
>> Subject: Re: troubles with HBase unit tests using MiniMRCluster on
>> 0.23.1-SNAPSHOT
>>
>> Hi Mahadev,
>>
>>> mvn -Dhadoop.profile=23 clean test
>>>   -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
>>> on hbase 0.92. (latest svn branch)  should be able to recreate this?
>>> (Assuming your patch applied on MR-3654)
>>
>> Yes, but as Hitesh noted on MAPREDUCE-3654, it seems the Yarn client
>> provider is not found or used. (I don't know yet how all of this works...)
>> So the issue with LocalClientProvider not liking the MASTER_ADDRESS set to
>> "test" by the mini Yarn cluster is merely a symptom. Likewise perhaps the
>> FNFE problem.
>>
>>> I can try that on my setup and see what the issue is.
>>
>> If you could spare a few minutes, that would be very helpful I think.
>> Perhaps our POM needs updating, some artifact is perhaps not being pulled in
>> for testing, and while this was fine for 0.23.0, that is no longer the case
>> for 0.23.1+?
>>
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>> ________________________________
>> From: Mahadev Konar <mahadev@hortonworks.com>
>> To: mapreduce-user@hadoop.apache.org; Andrew Purtell <apurtell@apache.org>
>> Sent: Tuesday, January 10, 2012 6:30 PM
>> Subject: Re: troubles with HBase unit tests using MiniMRCluster on
>> 0.23.1-SNAPSHOT
>>
>> Hi Andrew,
>> Thanks for filing MAPREDUCE-3654. Ill take a look. As for the
>> filenotfound issue
>> mvn -Dhadoop.profile=23 clean test
>> -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
>>
>> on hbase 0.92. (latest svn  branch)  should be able to recreate this?
>> (Assuming your patch applied on MR-3654)
>>
>> I can try that on my setup and see what the issue is.
>>
>> thanks
>> mahadev
>>
>> On Tue, Jan 10, 2012 at 6:11 PM, Andrew Purtell <apurtell@apache.org> wrote:
>>> Hi,
>>>
>>> I've been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but
>>> have run into a couple of issues. Perhaps I'm doing something wrong.
>>>
>>> What I've done:
>>>
>>>   - Checked out Hadoop branch-0.23 and installed artifacts into the local
>>> maven cache:
>>>
>>>      $ cd /path/to/hadoop
>>>      $ mvn clean install -DskipTests
>>>
>>>   - Checked out HBase branch-0.92 and attempted to test under the maven
>>> profile for 0.23:
>>>
>>>      $ cd /path/to/hbase
>>>      $ mvn -Dhadoop.profile=23 clean test
>>> -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
>>>
>>> The first issue I encountered was that the MiniMRCluster would not
>>> initialize:
>>>
>>> 2012-01-10 18:01:02,489 INFO  [main] mapreduce.TestTableMapReduce(153):
>>> Started mrtest
>>> 2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying
>>> ClientProtocolProvider :
>>> org.apache.hadoop.mapred.LocalClientProtocolProvider
>>> 2012-01-10 18:01:02,494 INFO  [main] mapreduce.Cluster(116): Failed to use
>>> org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
>>> "mapreduce.jobtracker.address" configuration value for LocalJobRunner :
>>> "test"
>>> 2012-01-10 18:01:02,494 ERROR [main] security.UserGroupInformation(1160):
>>> PriviledgedActionException as:apurtell (auth:SIMPLE)
>>> cause:java.io.IOException: Cannot initialize Cluster. Please check your
>>> configuration for mapreduce.framework.name and the correspond server
>>> addresses.
>>> 2012-01-10 18:01:02,497 INFO  [main] hbase.HBaseTestingUtility(1173):
>>> Stopping mini mapreduce cluster...
>>> 2
>>>
>>> I filed MAPREDUCE-3654
>>> (https://issues.apache.org/jira/browse/MAPREDUCE-3654). There could be
>>> more
>>> going on here, but the patch on that issue let me move forward.
>>>
>>> Now I am seeing this:
>>>
>>> 2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked
>>> org.apache.hadoop.mapred.LocalClientProtocolProvider as the
>>> ClientProtocolProvider
>>> 2012-01-10 17:51:35,243 INFO  [main] mapreduce.TableOutputFormat(197):
>>> Created table instance for mrtest
>>> 2012-01-10 17:51:35,249 DEBUG [main] mapreduce.JobSubmitter(347):
>>> Configuring job job_local_0001 with
>>>
>>> file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
>>> as the submit dir
>>> 2012-01-10 17:51:35,251 DEBUG [main] mapreduce.JobSubmitter(565): adding
>>> the
>>> following namenodes' delegation
>>> tokens:[hdfs://localhost.localdomain:54525]
>>> 2012-01-10 17:51:35,251 WARN  [main] mapreduce.JobSubmitter(139): Use
>>> GenericOptionsParser for parsing the arguments. Applications should
>>> implement Tool for the same.
>>> 2012-01-10 17:51:35,252 DEBUG [main] mapreduce.JobSubmitter(158): default
>>> FileSystem: file:///
>>> 2012-01-10 17:51:35,261 WARN  [main] mapreduce.JobSubmitter(241): No job
>>> jar
>>> file set.  User classes may not be found. See Job or Job#setJar(String).
>>> 2012-01-10 17:51:35,266 INFO  [main] mapreduce.JobSubmitter(388): Cleaning
>>> up the staging area
>>>
>>> file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
>>> 2012-01-10 17:51:35,268 ERROR [main] security.UserGroupInformation(1160):
>>> PriviledgedActionException as:apurtell (auth:SIMPLE)
>>> cause:java.io.FileNotFoundException: File does not exist:
>>>
>>> /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
>>> 2012-01-10 17:51:35,273 INFO  [main] hbase.HBaseTestingUtility(1173):
>>> Stopping mini mapreduce cluster...
>>>
>>> Of course that FileNotFoundException is for a file that does indeed exist,
>>> in the local maven cache:
>>>
>>>     $ ls -l
>>>
>>> /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
>>>     -rw-rw-r-- 1 apurtell apurtell 764555 2012-01-05 18:56
>>>
>>> /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
>>>
>>> I set permissions on all directories from /home/apurtell to the jar to
>>> 'rwxr-xr-x' just to make sure there was not some kind of odd permissions
>>> problem.
>>>
>>> Any idea what is going on here?
>>>
>>> Best regards,
>>>
>>> - Andy
>>>
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
>>
>>
>>
>>

Mime
View raw message