Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
References: <1326247901.44415.YahooMailNeo@web164512.mail.gq1.yahoo.com>
 <CALF7aMTrQzK-CQGBAAcYmVcAEdN6B66wHhrnanyGEkbUCZ4HvQ@mail.gmail.com>
 <1326304346.97036.YahooMailNeo@web164510.mail.gq1.yahoo.com>
 <1326410633.67784.YahooMailNeo@web164508.mail.gq1.yahoo.com>
 <CALF7aMR+d0TdZb=GcCnjqSx2z=7u4m2_2khxQyAWVLeweLHvDQ@mail.gmail.com>
 <CALF7aMSXdFXBeyLqgLe5guDehOGJ+iJqARs0m8q10f7-MGbfNg@mail.gmail.com>
 <CALte62zxbPdLaVOYUV=K57KzL-jyz=q9-DB3UNbMNvhCMTp5gQ@mail.gmail.com>
 <1326567333.38547.YahooMailNeo@web164502.mail.gq1.yahoo.com>
Message-ID: <1326568752.43230.YahooMailNeo@web164516.mail.gq1.yahoo.com>
Date: Sat, 14 Jan 2012 11:19:12 -0800 (PST)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: Andrew Purtell <apurtell@apache.org>
Subject: Re: troubles with HBase unit tests using MiniMRCluster on
 0.23.1-SNAPSHOT
To: Ted Yu <yuzhihong@gmail.com>,
  "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Cc: Stack <stack@duboce.net>
In-Reply-To: <1326567333.38547.YahooMailNeo@web164502.mail.gq1.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Mahadev,=0A=0A=0AAs for the fixes to get the HBase 0.92.x unit tests wor=
king with Hadoop 0.23.x, we can fix our POM in 0.92.1, no problem. My thank=
s to you and Hitesh. I'd like to be running 0.92 under 0.23 on test cluster=
s soon. Successful unit testing is the first step.=0A=0A=0ABest regards,=0A=
=0A=0A=A0=A0=A0 - Andy=0A=0AProblems worthy of attack prove their worth by =
hitting back. - Piet Hein (via Tom White)=0A=0A>=0A>>______________________=
__________=0A>> From: Ted Yu <yuzhihong@gmail.com>=0A>>To: mapreduce-user@h=
adoop.apache.org =0A>>Cc: Andrew Purtell <apurtell@apache.org>; Stack <stac=
k@duboce.net> =0A>>Sent: Thursday, January 12, 2012 8:10 PM=0A>>Subject: Re=
: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT=0A>=
> =0A>>=0A>>Mahadev:=0A>>I logged https://issues.apache.org/jira/browse/HBA=
SE-5191=0A>>=0A>>I suggest fixing it on HBase TRUNK.=0A>>=0A>>Thanks=0A>>=
=0A>>=0A>>On Thu, Jan 12, 2012 at 7:53 PM, Mahadev Konar <mahadev@hortonwor=
ks.com> wrote:=0A>>=0A>>Andrew,=0A>>>=A0I just checked out 0.92 branch and =
tried running:=0A>>>=0A>>>=0A>>>mvn -Dhadoop.profile=3D23 clean test=0A>>>-=
Dtest=3Dorg.apache.hadoop.hbase.mapreduce.TestTableMapReduce=0A>>>=0A>>>Loo=
ks like a compilation issue:=0A>>>=0A>>>------------=0A>>>[ERROR] Failed to=
 execute goal=0A>>>org.apache.maven.plugins:maven-compiler-plugin:2.0.2:tes=
tCompile=0A>>>(default-testCompile) on project hbase: Compilation failure=
=0A>>>[ERROR] /Users/mahadev/workspace/hbase-workspace/hbase-git/src/test/j=
ava/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java:[341,33]=
=0A>>>cannot find symbol=0A>>>[ERROR] symbol =A0: variable dnRegistration=
=0A>>>[ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataNo=
de=0A>>>[ERROR] -> [Help 1]=0A>>>[ERROR]=0A>>>--------------=0A>>>=0A>>>Loo=
ks like this commit caused the issue:=0A>>>=0A>>>=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A>>>commit 26eb39b424838717cf87833=
2db8c837b02117c5b=0A>>>=A0 =A0HBASE-5163 =A0TestLogRolling#testLogRollOnDat=
anodeDeath fails=0A>>>sometimes on Jenkins or hadoop QA=0A>>>=A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0 ("The directory is already locked.") (N Keywal)=0A>=
>>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A>>>=0A>>>Reve=
rting that patch locally, we have a fix to get the tests working=0A>>>(than=
ks to Hitesh). Mostly changes to pom files. Should we open a=0A>>>HBase jir=
a for the compilation issue? I suppose 0.92 wont be the=0A>>>target fix or =
would it?=0A>>>=0A>>>thanks=0A>>>mahadev=0A>>>=0A>>>=0A>>>On Thu, Jan 12, 2=
012 at 6:02 PM, Mahadev Konar <mahadev@hortonworks.com> wrote:=0A>>>> HI An=
drew,=0A>>>> =A0Sorry got side tracked with something else. Taking a look n=
ow.=0A>>>>=0A>>>> mahadev=0A>>>>=0A>>>> On Thu, Jan 12, 2012 at 3:23 PM, An=
drew Purtell <apurtell@apache.org> wrote:=0A>>>>> Hi Mahadev,=0A>>>>>=0A>>>=
>> Was this reproducible?=0A>>>>>=0A>>>>> Best regards,=0A>>>>>=0A>>>>> =A0=
 - Andy=0A>>>>>=0A>>>>> Problems worthy of attack prove their worth by hitt=
ing back. - Piet Hein=0A>>>>> (via Tom White)=0A>>>>>=0A>>>>> _____________=
___________________=0A>>>>> From: Andrew Purtell <apurtell@apache.org>=0A>>=
>>> To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.or=
g>=0A>>>>> Cc: Mahadev Konar <mahadev@hortonworks.com>=0A>>>>> Sent: Wednes=
day, January 11, 2012 9:52 AM=0A>>>>>=0A>>>>> Subject: Re: troubles with HB=
ase unit tests using MiniMRCluster on=0A>>>>> 0.23.1-SNAPSHOT=0A>>>>>=0A>>>=
>> Hi Mahadev,=0A>>>>>=0A>>>>>> mvn -Dhadoop.profile=3D23 clean test=0A>>>>=
>>=A0=A0 -Dtest=3Dorg.apache.hadoop.hbase.mapreduce.TestTableMapReduce=0A>>=
>>>> on hbase 0.92. (latest svn branch)=A0 should be able to recreate this?=
=0A>>>>>> (Assuming your patch applied on MR-3654)=0A>>>>>=0A>>>>> Yes, but=
 as Hitesh noted on MAPREDUCE-3654, it seems the Yarn client=0A>>>>> provid=
er is not found or used. (I don't know yet how all of this works...)=0A>>>>=
> So the issue with LocalClientProvider not liking the MASTER_ADDRESS set t=
o=0A>>>>> "test" by the mini Yarn cluster is merely a symptom. Likewise per=
haps the=0A>>>>> FNFE problem.=0A>>>>>=0A>>>>>> I can try that on my setup =
and see what the issue is.=0A>>>>>=0A>>>>> If you could spare a few minutes=
, that would be very helpful I think.=0A>>>>> Perhaps our POM needs updatin=
g, some artifact is perhaps not being pulled in=0A>>>>> for testing, and wh=
ile this was fine for 0.23.0, that is no longer the case=0A>>>>> for 0.23.1=
+?=0A>>>>>=0A>>>>> Best regards,=0A>>>>>=0A>>>>> =A0=A0 - Andy=0A>>>>>=0A>>=
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hei=
n=0A>>>>> (via Tom White)=0A>>>>>=0A>>>>> ________________________________=
=0A>>>>> From: Mahadev Konar <mahadev@hortonworks.com>=0A>>>>> To: mapreduc=
e-user@hadoop.apache.org; Andrew Purtell <apurtell@apache.org>=0A>>>>> Sent=
: Tuesday, January 10, 2012 6:30 PM=0A>>>>> Subject: Re: troubles with HBas=
e unit tests using MiniMRCluster on=0A>>>>> 0.23.1-SNAPSHOT=0A>>>>>=0A>>>>>=
 Hi Andrew,=0A>>>>> Thanks for filing MAPREDUCE-3654. Ill take a look. As f=
or the=0A>>>>> filenotfound issue=0A>>>>> mvn -Dhadoop.profile=3D23 clean t=
est=0A>>>>> -Dtest=3Dorg.apache.hadoop.hbase.mapreduce.TestTableMapReduce=
=0A>>>>>=0A>>>>> on hbase 0.92. (latest svn=A0 branch)=A0 should be able to=
 recreate this?=0A>>>>> (Assuming your patch applied on MR-3654)=0A>>>>>=0A=
>>>>> I can try that on my setup and see what the issue is.=0A>>>>>=0A>>>>>=
 thanks=0A>>>>> mahadev=0A>>>>>=0A>>>>> On Tue, Jan 10, 2012 at 6:11 PM, An=
drew Purtell <apurtell@apache.org> wrote:=0A>>>>>> Hi,=0A>>>>>>=0A>>>>>> I'=
ve been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but=0A>=
>>>>> have run into a couple of issues. Perhaps I'm doing something wrong.=
=0A>>>>>>=0A>>>>>> What I've done:=0A>>>>>>=0A>>>>>> =A0 - Checked out Hado=
op branch-0.23 and installed artifacts into the local=0A>>>>>> maven cache:=
=0A>>>>>>=0A>>>>>> =A0=A0=A0=A0 $ cd /path/to/hadoop=0A>>>>>> =A0=A0=A0=A0 =
$ mvn clean install -DskipTests=0A>>>>>>=0A>>>>>> =A0 - Checked out HBase b=
ranch-0.92 and attempted to test under the maven=0A>>>>>> profile for 0.23:=
=0A>>>>>>=0A>>>>>> =A0=A0=A0=A0 $ cd /path/to/hbase=0A>>>>>> =A0=A0=A0=A0 $=
 mvn -Dhadoop.profile=3D23 clean test=0A>>>>>> -Dtest=3Dorg.apache.hadoop.h=
base.mapreduce.TestTableMapReduce=0A>>>>>>=0A>>>>>> The first issue I encou=
ntered was that the MiniMRCluster would not=0A>>>>>> initialize:=0A>>>>>>=
=0A>>>>>> 2012-01-10 18:01:02,489 INFO=A0 [main] mapreduce.TestTableMapRedu=
ce(153):=0A>>>>>> Started mrtest=0A>>>>>> 2012-01-10 18:01:02,493 DEBUG [ma=
in] mapreduce.Cluster(93): Trying=0A>>>>>> ClientProtocolProvider :=0A>>>>>=
> org.apache.hadoop.mapred.LocalClientProtocolProvider=0A>>>>>> 2012-01-10 =
18:01:02,494 INFO=A0 [main] mapreduce.Cluster(116): Failed to use=0A>>>>>> =
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid=
=0A>>>>>> "mapreduce.jobtracker.address" configuration value for LocalJobRu=
nner :=0A>>>>>> "test"=0A>>>>>> 2012-01-10 18:01:02,494 ERROR [main] securi=
ty.UserGroupInformation(1160):=0A>>>>>> PriviledgedActionException as:apurt=
ell (auth:SIMPLE)=0A>>>>>> cause:java.io.IOException: Cannot initialize Clu=
ster. Please check your=0A>>>>>> configuration for mapreduce.framework.name=
 and the correspond server=0A>>>>>> addresses.=0A>>>>>> 2012-01-10 18:01:02=
,497 INFO=A0 [main] hbase.HBaseTestingUtility(1173):=0A>>>>>> Stopping mini=
 mapreduce cluster...=0A>>>>>> 2=0A>>>>>>=0A>>>>>> I filed MAPREDUCE-3654=
=0A>>>>>> (https://issues.apache.org/jira/browse/MAPREDUCE-3654). There cou=
ld be=0A>>>>>> more=0A>>>>>> going on here, but the patch on that issue let=
 me move forward.=0A>>>>>>=0A>>>>>> Now I am seeing this:=0A>>>>>>=0A>>>>>>=
 2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked=0A>>>>=
>> org.apache.hadoop.mapred.LocalClientProtocolProvider as the=0A>>>>>> Cli=
entProtocolProvider=0A>>>>>> 2012-01-10 17:51:35,243 INFO=A0 [main] mapredu=
ce.TableOutputFormat(197):=0A>>>>>> Created table instance for mrtest=0A>>>=
>>> 2012-01-10 17:51:35,249 DEBUG [main] mapreduce.JobSubmitter(347):=0A>>>=
>>> Configuring job job_local_0001 with=0A>>>>>>=0A>>>>>> file:/tmp/hadoop-=
apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001=0A>>>>>>=
 as the submit dir=0A>>>>>> 2012-01-10 17:51:35,251 DEBUG [main] mapreduce.=
JobSubmitter(565): adding=0A>>>>>> the=0A>>>>>> following namenodes' delega=
tion=0A>>>>>> tokens:[hdfs://localhost.localdomain:54525]=0A>>>>>> 2012-01-=
10 17:51:35,251 WARN=A0 [main] mapreduce.JobSubmitter(139): Use=0A>>>>>> Ge=
nericOptionsParser for parsing the arguments. Applications should=0A>>>>>> =
implement Tool for the same.=0A>>>>>> 2012-01-10 17:51:35,252 DEBUG [main] =
mapreduce.JobSubmitter(158): default=0A>>>>>> FileSystem: file:///=0A>>>>>>=
 2012-01-10 17:51:35,261 WARN=A0 [main] mapreduce.JobSubmitter(241): No job=
=0A>>>>>> jar=0A>>>>>> file set.=A0 User classes may not be found. See Job =
or Job#setJar(String).=0A>>>>>> 2012-01-10 17:51:35,266 INFO=A0 [main] mapr=
educe.JobSubmitter(388): Cleaning=0A>>>>>> up the staging area=0A>>>>>>=0A>=
>>>>> file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/=
job_local_0001=0A>>>>>> 2012-01-10 17:51:35,268 ERROR [main] security.UserG=
roupInformation(1160):=0A>>>>>> PriviledgedActionException as:apurtell (aut=
h:SIMPLE)=0A>>>>>> cause:java.io.FileNotFoundException: File does not exist=
:=0A>>>>>>=0A>>>>>> /home/apurtell/.m2/repository/org/apache/zookeeper/zook=
eeper/3.4.2/zookeeper-3.4.2.jar=0A>>>>>> 2012-01-10 17:51:35,273 INFO=A0 [m=
ain] hbase.HBaseTestingUtility(1173):=0A>>>>>> Stopping mini mapreduce clus=
ter...=0A>>>>>>=0A>>>>>> Of course that FileNotFoundException is for a file=
 that does indeed exist,=0A>>>>>> in the local maven cache:=0A>>>>>>=0A>>>>=
>> =A0=A0=A0 $ ls -l=0A>>>>>>=0A>>>>>> /home/apurtell/.m2/repository/org/ap=
ache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar=0A>>>>>> =A0=A0=A0 -rw-r=
w-r-- 1 apurtell apurtell 764555 2012-01-05 18:56=0A>>>>>>=0A>>>>>> /home/a=
purtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2=
.jar=0A>>>>>>=0A>>>>>> I set permissions on all directories from /home/apur=
tell to the jar to=0A>>>>>> 'rwxr-xr-x' just to make sure there was not som=
e kind of odd permissions=0A>>>>>> problem.=0A>>>>>>=0A>>>>>> Any idea what=
 is going on here?=0A>>>>>>=0A>>>>>> Best regards,=0A>>>>>>=0A>>>>>> - Andy=
=0A>>>>>>=0A>>>>>>=0A>>>>>> Problems worthy of attack prove their worth by =
hitting back. - Piet Hein=0A>>>>>> (via Tom White)=0A>>>>>=0A>>>>>=0A>>>>>=
=0A>>>>>=0A>>>=0A>>=0A>>=0A>>=0A>=0A>