hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Lilley <john.lil...@redpoint.net>
Subject RE: Hadoop JARs and Eclipse
Date Thu, 06 Jun 2013 12:21:37 GMT
Thanks so much for your thorough explanation.

-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com] 
Sent: Wednesday, June 05, 2013 8:18 PM
To: <user@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse

Hi John,

On Thu, Jun 6, 2013 at 1:21 AM, John Lilley <john.lilley@redpoint.net> wrote:
> -- From where will it fetch the Hadoop JARs?

>From the Maven Central repository (we publish our jars and dependencies are also available
there), or a custom defined repository if you lack internet access.

> -- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead
of those from the official release site?

If you built Hadoop locally with "mvn install", then their version is added to your local
repository (~/.m2) and can be "referred' to after a successful "mvn install".

> -- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate
source files and display them if I need to step in?

No, see below.

> -- Step into the source code of referenced Hadoop code in the debugger.

This one is easy with the Eclipse Maven IDE plugin, see http://stackoverflow.com/questions/310720/get-source-jar-files-attached-to-eclipse-for-maven-managed-dependencies.
This lets you view and also debug-step into sources of dependencies.

> -- Make changes to the Hadoop source and rebuild it if things are not working as expected
(e.g. to add logging).
> -- Switch to building from SVN instead of a release snapshot, and apply patches.

If you're wanting to use the branch-2 tip or the trunk tip, then they carry their own version
(2.1.0-beta-SNAPSHOT, or 3.0.0-SNAPSHOT and such). Referring these versions in the pom.xml
of your project would be the first step (you may also have to enable discovery of snapshot
releases?). Second would be to make the changes to the hadoop source and running a global
"mvn install" to update the local repository
(~/.m2) versions to the patched ones. Subsequently, your project will be able to find it.

What am saying is you can do this without having to manage Hadoop as an eclipse project -
just patch and build (plus install) on CLI, but mark your project to reference their exact
SNAPSHOT version so it discovers the new built jars.

Harsh J

View raw message