hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8009) Create hadoop-client and hadoop-minicluster artifacts for downstream projects
Date Mon, 06 Feb 2012 16:20:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201366#comment-13201366

Matt Foley commented on HADOOP-8009:

Hi, I'm okay in principle with adding something to branch-1.0 for this, but it doesn't look
like either of the two patches currently posted are appropriate for adding directly to branch-1.0.
 I currently see the following issues: \\
1. integration into the code branch \\
1a) directory layout \\
1b) ant vs maven \\
2. need for top-level pom \\
3. automatically picking up the version number so we don't have to edit it in every release

To address 1a and 2, would the following make sense?
* at the top level of the branch-1.0 code tree we add a directory named "integration-artifacts/".
* under it, put "pom.xml", "hadoop-client/pom.xml", and "hadoop-minicluster/pom.xml"
This avoids putting a "pom.xml" at the top level of branch-1.0, when the majority of 1.0 is
built with ant not maven.

To address 1b and 3,
* provide an ant build rule in build.xml for target name "intartifactdeploy" which passes
the build version number to an invocation of maven upon "integration-artifacts/pom.xml"
* add "intartifactdeploy" as a dependency to the existing ant target "mvn-deploy"

If this makes sense, and you are willing to put together the appropriate patch, I'll include
it in branch-1.0 and branch-1.

> Create hadoop-client and hadoop-minicluster artifacts for downstream projects 
> ------------------------------------------------------------------------------
>                 Key: HADOOP-8009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8009
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0, 0.24.0, 0.23.1, 1.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.1
>         Attachments: HADOOP-8009-existing-releases.patch, HADOOP-8009.patch
> Using Hadoop from projects like Pig/Hive/Sqoop/Flume/Oozie or any in-house system that
interacts with Hadoop is quite challenging for the following reasons:
> * *Different versions of Hadoop produce different artifacts:* Before Hadoop 0.23 there
was a single artifact hadoop-core, starting with Hadoop 0.23 there are several (common, hdfs,
mapred*, yarn*)
> * *There are no 'client' artifacts:* Current artifacts include all JARs needed to run
the services, thus bringing into clients several JARs that are not used for job submission/monitoring
(servlet, jsp, tomcat, jersey, etc.)
> * *Doing testing on the client side is also quite challenging as more artifacts have
to be included than the dependencies define:* for example, the history-server artifact has
to be explicitly included. If using Hadoop 1 artifacts, jersey-server has to be explicitly
> * *3rd party dependencies change in Hadoop from version to version:* This makes things
complicated for projects that have to deal with multiple versions of Hadoop as their exclusions
list become a huge mix & match of artifacts from different Hadoop versions and it may
be break things when a particular version of Hadoop requires a dependency that other version
of Hadoop does not require.
> Because of this it would be quite convenient to have the following 'aggregator' artifacts:
> * *org.apache.hadoop:hadoop-client* : it includes all required JARs to use Hadoop client
APIs (excluding all JARs that are not needed for it)
> * *org.apache.hadoop:hadoop-test* : it includes all required JARs to run Hadoop Mini
> These aggregator artifacts would be created for current branches under development (trunk,
0.22, 0.23, 1.0) and for released versions that are still in use.
> For branches under development, these artifacts would be generated as part of the build.
> For released versions we would have a a special branch used only as vehicle for publishing
the corresponding 'aggregator' artifacts.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message