Mailing-List: contact dev-help@pig.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@pig.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAJ8L3biQkoE1fMz=gX3iXv6R3DgiV=JRP3q-1DJ4i-GFtbRvOg@mail.gmail.com>
References: 
 <CA+RK=_BEjZGJWWtgfBBURbvfoEW440tFw7_CDPCpxdc-xyGUhQ@mail.gmail.com>
 <CAJ8L3biQkoE1fMz=gX3iXv6R3DgiV=JRP3q-1DJ4i-GFtbRvOg@mail.gmail.com>
From: Andrew Purtell <apurtell@apache.org>
Date: Sat, 11 May 2013 09:19:29 +0800
Message-ID: 
 <CA+RK=_D=rOU+rq9VnHghfc+ZxDnHuJUBHH4tSOwb6vat-b+Usg@mail.gmail.com>
Subject: Re: Unit test classpath trouble
To: dev@pig.apache.org
Content-Type: multipart/alternative; boundary=bcaec548a4f19e33df04dc6715d8

--bcaec548a4f19e33df04dc6715d8
Content-Type: text/plain; charset=ISO-8859-1

I've tried that, thanks. I did a bit more investigation and it seems the
issue is recent Hadoop 2 releases. Has anyone tried running Pig unit tests
using a more recent Hadoop release than 2.0.0-alpha? Maybe my trouble is a
simple thing that someone with more experience with Pig internals would see
right away? Cluster testing seems ok. It's just unit tests that fail. But
that is concerning.

I'm trying HEAD of branch-0.11.

My Java is version "1.6.0_43" Java(TM) SE Runtime Environment (build
1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed
mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).

With defaults and only -Dhadoopversion=23 on the Ant command line, it seems
ok.

With build.properties of:

hadoopversion=23
hadoop-common.version=2.0.4-alpha
hadoop-hdfs.version=2.0.4-alpha
hadoop-mapreduce.version=2.0.4-alpha


or defined on the Ant command line, I'll see unit test failures like:

Testcase: testAccumWithDistinct took 0.868 sec
        Caused an ERROR
org/apache/hadoop/mapred/ResourceMgrDelegate
java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/ResourceMgrDelegate
        at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
        at
org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
        at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
        at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
        at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
        at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
        at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
        at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
        at org.apache.pig.PigServer.storeEx(PigServer.java:931)
        at org.apache.pig.PigServer.store(PigServer.java:898)
        at org.apache.pig.PigServer.openIterator(PigServer.java:811)
        at
org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424)

That suggests a cause but I've not started spelunking code with the hope
this is something simple that someone has already encountered.


On Sat, May 11, 2013 at 1:31 AM, Johnny Zhang <xiaoyuz@cloudera.com> wrote:

> Hi, Andrew:
> Does something like "-Dhadoopversion=23" help ? eg. ant clean test
> -Dhadoopversion=23 -Dtest.junit.output.format=xml
>
> Johnny
>
>
> On Fri, May 10, 2013 at 3:39 AM, Andrew Purtell <apurtell@apache.org>
> wrote:
>
> > Please pardon the basic question. I'm building Pig 0.11.2-SNAPSHOT
> against
> > Hadoop 2.0.4. 'ant package' and full cluster tests work fine, but I'm not
> > having much luck with running the unit tests, 'ant test-core' or 'ant
> > test'. The problem looks to be a MR app classpath issue.
> >
> > Sometimes: java.lang.NoClassDefFoundError:
> > org/apache/hadoop/yarn/client/YarnClientImpl
> >
> > Sometimes: java.lang.NoClassDefFoundError:
> > org/apache/hadoop/mapred/ResourceMgrDelegate
> >
> > A few Google searches have turned up no useful pointers. Maybe there is
> > something simple I am missing? How do you set up for running unit tests
> on
> > your dev boxes?
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

--bcaec548a4f19e33df04dc6715d8--