Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.210.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAH2Q5xeBnOXObrL6i9XQ90Vb7KTp5z8wZk=mLT4t3WFtB373vg@mail.gmail.com>
References: 
 <CAH2Q5xeBnOXObrL6i9XQ90Vb7KTp5z8wZk=mLT4t3WFtB373vg@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Thu, 20 Dec 2012 11:31:09 +0530
Message-ID: 
 <CAOcnVr2o2CuhuMadSUsgLzfdrj5iR9n=XrB0Ut_KuW4n7=w5iQ@mail.gmail.com>
Subject: Re: Development against hadoop v 0.23.5
To: virappan@gmail.com
Cc: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Hi Anil,

Usage oriented questions should be directed at user@hadoop.apache.org.
Also, you are running into an MR2 issue (not related to YARN - which
is just a platform MR2 runs on. See [1]).

I've moved this to the proper thread (bcc'd yarn-dev). My replies inline.

On Thu, Dec 20, 2012 at 6:06 AM, anil chaurasia <virappan@gmail.com> wrote:
> Hi All,
>
> I have a small prototype application that was running against 1.0.X version
> of hadoop.
> We wanted to try out the hadoop-yarn to get some number using the yarn
> release and setup a single node hadoop system.
>
> When I run the job with hadoop 1.0.X it works fine but when I try to run it
> against the 0.23.5 I get following error:
>
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.JobContext, but class was expected

Upgrading jobs from 1.x to 2.x (or 0.23.x), requires that your jobs be
recompiled against the new version, cause there are incompatible
changes in this upgrade that a recompilation helps catch and also
resolve.

> I am sure that there are significant changes to the hadoop framework in
> YARN release.
> So I decided to change the code as per the new interface.

I don't think you need to change much code as the APIs are still the
same. A recompilation against proper new dependencies should do the
trick.

> But when I try to use the maven to import the dependecies, maven complains
> about the artifacts. ( The maven repo only has pom.xml files and no jar
> files are included . )

Can you share your <dependencies>?

In 1.x you may have had "hadoop-core" as one dependency. In 2.x (or
0.23.x, a subset), you have several broken out components so we
provide a wrapper dependency called "hadoop-client" which should be
all that you need to include. There is no more "hadoop-core" in 2.x
(or 0.23.x).

> I was using maven dependency hadoop-core for ( 1.0.X) and it was working so
> I though I would get the yarn-core/yarn-project/hadoop-yarn but all of them
> had missing artifacts.
>
> One thing I did notice that the other yarn modules such as
> hadoop-yarn-common did have artifcats.

The 0.23.5 artifacts are all available on the Apache Maven
repositories http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/0.23.5/

> So, I wanted to know what are all the modules required to build a hadoop
> application using hadoop-yarn.

Any regular hadoop application may simply include hadoop-client to
pull in all dependencies.

P.s. You aren't exactly writing an YARN app, but an MR2 job (MR is an
app, you do not need to write that as its provided to you already).
[1] again.

> Is there a document on how to write a sample app using yarn somewhere on
> the wiki ? IF there is no wiki,  Could someone please point me to some
> documents where I can find information on how to write a new app using
> hadoop-yarn.

If you are interested in writing a distributed application (which is
not the same as writing an MR job), you can read [2].

[1] - http://www.cloudera.com/blog/2012/10/mr2-and-yarn-briefly-explained/
[2] - http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html


--
Harsh J