Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Wed, 15 Feb 2012 18:19:04 +0000 (UTC)
From: "Alejandro Abdelnur (Updated) (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: 
 <1652474090.41590.1329329944184.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1815534258.12105.1328031550286.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-8009) Create hadoop-client and
 hadoop-minicluster artifacts for downstream projects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HADOOP-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-8009:
---------------------------------------

    Attachment: HADOOP-8009-branch-0_22.patch

@Konstantin, attached is a patch for the 0.22 branch. I've test it installing the deploying to a local repo. I think I've got the signing cover, but you may have to tweak that if it does not work.

Also, on another note, 0.22 is not publishing the hadoop-streaming JAR to maven repo, because of that we cannot use 0.22 from Oozie. I'll open a JIRA for that.

Thanks.

                
> Create hadoop-client and hadoop-minicluster artifacts for downstream projects 
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-8009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8009
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0, 0.24.0, 0.23.1, 1.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.1, 1.0.1
>
>         Attachments: HADOOP-8009-branch-0_22.patch, HADOOP-8009-branch-1-add.patch, HADOOP-8009-branch-1.patch, HADOOP-8009-existing-releases.patch, HADOOP-8009.patch
>
>
> Using Hadoop from projects like Pig/Hive/Sqoop/Flume/Oozie or any in-house system that interacts with Hadoop is quite challenging for the following reasons:
> * *Different versions of Hadoop produce different artifacts:* Before Hadoop 0.23 there was a single artifact hadoop-core, starting with Hadoop 0.23 there are several (common, hdfs, mapred*, yarn*)
> * *There are no 'client' artifacts:* Current artifacts include all JARs needed to run the services, thus bringing into clients several JARs that are not used for job submission/monitoring (servlet, jsp, tomcat, jersey, etc.)
> * *Doing testing on the client side is also quite challenging as more artifacts have to be included than the dependencies define:* for example, the history-server artifact has to be explicitly included. If using Hadoop 1 artifacts, jersey-server has to be explicitly included.
> * *3rd party dependencies change in Hadoop from version to version:* This makes things complicated for projects that have to deal with multiple versions of Hadoop as their exclusions list become a huge mix & match of artifacts from different Hadoop versions and it may be break things when a particular version of Hadoop requires a dependency that other version of Hadoop does not require.
> Because of this it would be quite convenient to have the following 'aggregator' artifacts:
> * *org.apache.hadoop:hadoop-client* : it includes all required JARs to use Hadoop client APIs (excluding all JARs that are not needed for it)
> * *org.apache.hadoop:hadoop-minicluster* : it includes all required JARs to run Hadoop Mini Clusters
> These aggregator artifacts would be created for current branches under development (trunk, 0.22, 0.23, 1.0) and for released versions that are still in use.
> For branches under development, these artifacts would be generated as part of the build.
> For released versions we would have a a special branch used only as vehicle for publishing the corresponding 'aggregator' artifacts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira