hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elek, Marton (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDDS-447) separate ozone-dist and hadoop-dist projects with real classpath separation
Date Sun, 16 Sep 2018 21:30:00 GMT

     [ https://issues.apache.org/jira/browse/HDDS-447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Elek, Marton updated HDDS-447:
------------------------------
    Description: 
Currently we have the same hadoop-dist project to create both the ozone and and the hadoop
distribution.

To decouple ozone and hadoop build it would be great to create two different dist project.

The hadoop-dist should be cloned to hadoop-ozone/dist and from hadoop-dist/pom.xml we can
remove the hdds/ozone related items and from hadoop-ozone/dist/pom.xml we can remove the core
hadoop related part.

An other issue with the current distribution schema is the lack of real classpath separation.


The current hadoop distribution model is defined in the hadoop-project-dist which is parent
of all the component projects and the output of the distribution generation will be copied
by the dist-layout-stitching. There is no easy way to use command specific classpath as the
classpath is defined in component level (hdfs/yarn/mapreduce).

With this approach we will have a lot of unnecessary dependencies on the classpath (which
were not on the classpath at the time of the unit tests) and it's not possible (as an example)
use different type of jaxrs stack for different services (s3gateway vs scm).

As a simplified but more effective approach I propose to use the following method:

1. don't use hadoop-project-dist for ozone projects any more
2. During the build generate a classpath descriptor (with the dependency:build-classpath maven
plugin/goal) for all the projects
3. During the distribution copy all the required dependencies (with dependency:copy maven
plugin/goal) to a lib folder (share/ozone/lib)
4. During the distribution copy all the classpath descriptors to the classpath folder (share/ozone/classpath)
5. Put only the required jar files to the classpath with reading the classpath descriptor




  was:
Currently we have the same hadoop-dist project to create both the ozone and and the hadoop
distribution.

To decouple ozone and hadoop build it would be great to create two different dist project.

It could be done easily without modifying the current distribution schema:

The hadoop-dist should be cloned to hadoop-ozone/dist and from hadoop-dist/pom.xml we can
remove the hdds/ozone related items and from hadoop-ozone/dist/pom.xml we can remove the core
hadoop related part.


> separate ozone-dist and hadoop-dist projects with real classpath separation
> ---------------------------------------------------------------------------
>
>                 Key: HDDS-447
>                 URL: https://issues.apache.org/jira/browse/HDDS-447
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>            Reporter: Elek, Marton
>            Assignee: Elek, Marton
>            Priority: Major
>
> Currently we have the same hadoop-dist project to create both the ozone and and the hadoop
distribution.
> To decouple ozone and hadoop build it would be great to create two different dist project.
> The hadoop-dist should be cloned to hadoop-ozone/dist and from hadoop-dist/pom.xml we
can remove the hdds/ozone related items and from hadoop-ozone/dist/pom.xml we can remove the
core hadoop related part.
> An other issue with the current distribution schema is the lack of real classpath separation.

> The current hadoop distribution model is defined in the hadoop-project-dist which is
parent of all the component projects and the output of the distribution generation will be
copied by the dist-layout-stitching. There is no easy way to use command specific classpath
as the classpath is defined in component level (hdfs/yarn/mapreduce).
> With this approach we will have a lot of unnecessary dependencies on the classpath (which
were not on the classpath at the time of the unit tests) and it's not possible (as an example)
use different type of jaxrs stack for different services (s3gateway vs scm).
> As a simplified but more effective approach I propose to use the following method:
> 1. don't use hadoop-project-dist for ozone projects any more
> 2. During the build generate a classpath descriptor (with the dependency:build-classpath
maven plugin/goal) for all the projects
> 3. During the distribution copy all the required dependencies (with dependency:copy maven
plugin/goal) to a lib folder (share/ozone/lib)
> 4. During the distribution copy all the classpath descriptors to the classpath folder
(share/ozone/classpath)
> 5. Put only the required jar files to the classpath with reading the classpath descriptor




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message