hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-6671) To use maven for hadoop common builds
Date Fri, 13 May 2011 23:50:48 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: mvn-layout.sh
                hadoop-commons-maven.patch

I spent a few hours playing with hadoop-common mavenization following the ideas in the JIRA.

I've got it to a point that compiles java and native code, runs the tests (with native code
if present) and generates the JAR.

I've only had to modify 1 java file, one testcase that had 'build/classes' hardcoded to look
for a properties file.

I've had to move directories/files around to follow Maven dir layout.

Still to be done is javadocs, documentation, packaging (that would be an assembly descriptor),
wiring jdiff/clover/findbugs, and deployment configuration of artifacts (JARs/SOs).

The end goal will be to generate a TARBALL identical to the one it is being generated today.


I don't expect all that to be much work.

Once hadoop-commons is done, the same could be done in hadoop-hdfs and hadoop-mapreduce. Also
the contrib/ stuff could go into its own Maven module. Finally a root Maven project could
be used to wire all the above projects and external dependencies versions would be defined
there in a dependencyManagement section.

The good thing is that it is not required to do all at once, we can do common, then hdfs,
then mapreduce and finally contrib.

Attached you'll find a script that moves dir/files around to the maven expected locations
and a patch containing the Maven pom.xml file, a native-build.xml (Ant file invoked from maven
to build native code) and the 1 line change to a testcase.

Instructions to test it:

* checkout hadoop-common trunk
* run the attached script from hadoop-common root dir
* apply the the patch
* Use Maven 3 to to build/test
** -Dcompile.native enables native compilation

IMPORTANT: I couldn't figure this out yet but there is some issue with javah when invoked
from Maven/Ant (javah is not being found in LINUX because Maven changes JAVA_HOME to JRE location).
TEMPORARY WORKAROUND: softlink JDK/lib/tools.jar in JRE/lib/ext/

Before I continue working on this I want to know if folks would be OK with moving forward
with this patch.


> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: build.png, hadoop-commons-maven.patch, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to
manage dependencies, publish artifacts and have one single xml file(POM) for dependency management
and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message