hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahadev Konar <maha...@hortonworks.com>
Subject Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Date Fri, 13 Jan 2012 02:02:15 GMT
HI Andrew,
 Sorry got side tracked with something else. Taking a look now.

mahadev

On Thu, Jan 12, 2012 at 3:23 PM, Andrew Purtell <apurtell@apache.org> wrote:
> Hi Mahadev,
>
> Was this reproducible?
>
> Best regards,
>
>   - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
> ________________________________
> From: Andrew Purtell <apurtell@apache.org>
> To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
> Cc: Mahadev Konar <mahadev@hortonworks.com>
> Sent: Wednesday, January 11, 2012 9:52 AM
>
> Subject: Re: troubles with HBase unit tests using MiniMRCluster on
> 0.23.1-SNAPSHOT
>
> Hi Mahadev,
>
>> mvn -Dhadoop.profile=23 clean test
>>   -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
>> on hbase 0.92. (latest svn branch)  should be able to recreate this?
>> (Assuming your patch applied on MR-3654)
>
> Yes, but as Hitesh noted on MAPREDUCE-3654, it seems the Yarn client
> provider is not found or used. (I don't know yet how all of this works...)
> So the issue with LocalClientProvider not liking the MASTER_ADDRESS set to
> "test" by the mini Yarn cluster is merely a symptom. Likewise perhaps the
> FNFE problem.
>
>> I can try that on my setup and see what the issue is.
>
> If you could spare a few minutes, that would be very helpful I think.
> Perhaps our POM needs updating, some artifact is perhaps not being pulled in
> for testing, and while this was fine for 0.23.0, that is no longer the case
> for 0.23.1+?
>
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
> ________________________________
> From: Mahadev Konar <mahadev@hortonworks.com>
> To: mapreduce-user@hadoop.apache.org; Andrew Purtell <apurtell@apache.org>
> Sent: Tuesday, January 10, 2012 6:30 PM
> Subject: Re: troubles with HBase unit tests using MiniMRCluster on
> 0.23.1-SNAPSHOT
>
> Hi Andrew,
> Thanks for filing MAPREDUCE-3654. Ill take a look. As for the
> filenotfound issue
> mvn -Dhadoop.profile=23 clean test
> -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
>
> on hbase 0.92. (latest svn  branch)  should be able to recreate this?
> (Assuming your patch applied on MR-3654)
>
> I can try that on my setup and see what the issue is.
>
> thanks
> mahadev
>
> On Tue, Jan 10, 2012 at 6:11 PM, Andrew Purtell <apurtell@apache.org> wrote:
>> Hi,
>>
>> I've been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but
>> have run into a couple of issues. Perhaps I'm doing something wrong.
>>
>> What I've done:
>>
>>   - Checked out Hadoop branch-0.23 and installed artifacts into the local
>> maven cache:
>>
>>      $ cd /path/to/hadoop
>>      $ mvn clean install -DskipTests
>>
>>   - Checked out HBase branch-0.92 and attempted to test under the maven
>> profile for 0.23:
>>
>>      $ cd /path/to/hbase
>>      $ mvn -Dhadoop.profile=23 clean test
>> -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
>>
>> The first issue I encountered was that the MiniMRCluster would not
>> initialize:
>>
>> 2012-01-10 18:01:02,489 INFO  [main] mapreduce.TestTableMapReduce(153):
>> Started mrtest
>> 2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying
>> ClientProtocolProvider :
>> org.apache.hadoop.mapred.LocalClientProtocolProvider
>> 2012-01-10 18:01:02,494 INFO  [main] mapreduce.Cluster(116): Failed to use
>> org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
>> "mapreduce.jobtracker.address" configuration value for LocalJobRunner :
>> "test"
>> 2012-01-10 18:01:02,494 ERROR [main] security.UserGroupInformation(1160):
>> PriviledgedActionException as:apurtell (auth:SIMPLE)
>> cause:java.io.IOException: Cannot initialize Cluster. Please check your
>> configuration for mapreduce.framework.name and the correspond server
>> addresses.
>> 2012-01-10 18:01:02,497 INFO  [main] hbase.HBaseTestingUtility(1173):
>> Stopping mini mapreduce cluster...
>> 2
>>
>> I filed MAPREDUCE-3654
>> (https://issues.apache.org/jira/browse/MAPREDUCE-3654). There could be
>> more
>> going on here, but the patch on that issue let me move forward.
>>
>> Now I am seeing this:
>>
>> 2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked
>> org.apache.hadoop.mapred.LocalClientProtocolProvider as the
>> ClientProtocolProvider
>> 2012-01-10 17:51:35,243 INFO  [main] mapreduce.TableOutputFormat(197):
>> Created table instance for mrtest
>> 2012-01-10 17:51:35,249 DEBUG [main] mapreduce.JobSubmitter(347):
>> Configuring job job_local_0001 with
>>
>> file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
>> as the submit dir
>> 2012-01-10 17:51:35,251 DEBUG [main] mapreduce.JobSubmitter(565): adding
>> the
>> following namenodes' delegation
>> tokens:[hdfs://localhost.localdomain:54525]
>> 2012-01-10 17:51:35,251 WARN  [main] mapreduce.JobSubmitter(139): Use
>> GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 2012-01-10 17:51:35,252 DEBUG [main] mapreduce.JobSubmitter(158): default
>> FileSystem: file:///
>> 2012-01-10 17:51:35,261 WARN  [main] mapreduce.JobSubmitter(241): No job
>> jar
>> file set.  User classes may not be found. See Job or Job#setJar(String).
>> 2012-01-10 17:51:35,266 INFO  [main] mapreduce.JobSubmitter(388): Cleaning
>> up the staging area
>>
>> file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
>> 2012-01-10 17:51:35,268 ERROR [main] security.UserGroupInformation(1160):
>> PriviledgedActionException as:apurtell (auth:SIMPLE)
>> cause:java.io.FileNotFoundException: File does not exist:
>>
>> /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
>> 2012-01-10 17:51:35,273 INFO  [main] hbase.HBaseTestingUtility(1173):
>> Stopping mini mapreduce cluster...
>>
>> Of course that FileNotFoundException is for a file that does indeed exist,
>> in the local maven cache:
>>
>>     $ ls -l
>>
>> /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
>>     -rw-rw-r-- 1 apurtell apurtell 764555 2012-01-05 18:56
>>
>> /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
>>
>> I set permissions on all directories from /home/apurtell to the jar to
>> 'rwxr-xr-x' just to make sure there was not some kind of odd permissions
>> problem.
>>
>> Any idea what is going on here?
>>
>> Best regards,
>>
>> - Andy
>>
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>
>
>
>

Mime
View raw message