hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3305) Publish hadoop-core to the apache repository with an appropriate POM file
Date Tue, 25 Nov 2008 16:55:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650624#action_12650624

Steve Loughran commented on HADOOP-3305:

I've had a quick flick through and I'm impressed by the effort that Giri has gone to here.
This looks like the basis for moving everything in the core to ivy

1. It would be good to have everything driven by a single master properties file, with each
project having the option to override them (via their own libraries.properties) but not requiring
them to. This makes it easy to push up every app to a new version of, say, log4j, by changing
one file, and it keeps things consistent. Giri's patch shows how inconsistent the projects
are regarding versions of things, and that just leads to trouble down the line

2. Does Chukwa depend on hadoop-core? If it does, there's a good case for the ivy config file
of hadoop-core to contain some specific configurations for jetty and jsp support, so that
Chukwa can pull them in without having to repeat them. This is what we do in smartfrog by
cross-referencing component packages: only one package is allowed to import a third party
library; everything else has to depend on that package. It works very well when you move to
RPM distribution as the same dependencies and ownership rules apply there.

3. We'd need to go through the ivy reports of everything and make sure that nothing is pulling
in transient dependencies you don't want. If they pull in transients you do want, it is safer
to declare them and the version you desire. commons-logging is a notorious source of problems
here; you should only ever depend on its "master" version to avoid stuff you dont need like
avalon-logkit and bits of JMX.

4. src/contrib/hdfsproxy/ivybuild.xml has hard coded version numbers in the build file. It
shoud be driven from the .properties file

I need to play with this some more, by patching a clean version of the source tree and seeing
how it goes. Its a good design, there's just a few more tweaks we need to get in there

> Publish hadoop-core to the apache repository with an appropriate POM file
> -------------------------------------------------------------------------
>                 Key: HADOOP-3305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3305
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.16.2, 0.16.3
>            Reporter: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-3305.patch, hadoop-core-0.16.2.pom, ivy-support-first-pass.zip,
ivysupport.zip, rmlib.sh
> To let people downstream build/test with hadoop, using Apache Ivy or Apache Maven2 to
pull it down, hadoop-core needs to be published to the apache repository with a .pom file
that lists its mandatory dependencies.
> In an automated build process, this means
> -having a template XML pom defining all included dependencies (and excluded transient
dependency artifacts)
> -having a property file driving version numbering of all artifacts
> -copying this template with property expansion to create the release POM file
> -public releases only: sticking this POM file up on people.apache.org in the right place,
along with the JAR and some .md5 checksums
> There's a risk that if the hadoop team dont do this, someone else will (as mahout are
doing under http://people.apache.org/~kalle/mahout/maven2/org/apache/hadoop/ )
> This is bad as hadoop end up fielding the support calls from someone elses files. 
> Before automating the process, existing hadoop-core JARs can be pushed out with hand-encoded
POM files. The repository police dont allow pom files ever to be changed, so supporting existing
releases (.16.2, 0.16.3 ... ) is a way of beta testing the POMs. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message