hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elek, Marton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-40) Separating packaging of Ozone/HDDS from the main Hadoop
Date Fri, 11 May 2018 09:55:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471708#comment-16471708
] 

Elek, Marton commented on HDDS-40:
----------------------------------

Thanks [~anu] to test it. I just got a new idea as I read your tests.

Until now the biggest problem was the usage of the version in the dist directory name (hadoop-dist/target/hadoop-3.2.0-SNAPSHOT).
To mount the dist package to a docker image we needed the exact version number (3.2.0-SNAPSHOT)
to assembly the right path.

The workaround was to copy the docker-compose files during the build with maven filtering
which replaced a place holder (${hadoop.version}) with the actual version.

But now, we can simplify it a lot as the version could be removed from the directory name
(instead of hadoop-dist/target/ozone-2.1.0-SNAPSHOT I would use hadoop-dist/target/ozone).
With this approach both the docker-compose files and the ozone acceptance tests could be simplified
as we don't need the version any more.

I will try it out and update the patch (and also fix the README)

> Separating packaging of Ozone/HDDS from the main Hadoop
> -------------------------------------------------------
>
>                 Key: HDDS-40
>                 URL: https://issues.apache.org/jira/browse/HDDS-40
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>            Reporter: Elek, Marton
>            Assignee: Elek, Marton
>            Priority: Major
>         Attachments: HDDS-40.001.patch
>
>
> According to the community vote, Ozone/Hdds release cycle should be independent from
the Hadoop release cycle.
> To make it possible we need a separated ozone package.
> *The current state:*
> We have just one output tar/directory under hadoop-dist (hadoop-3.2.0). It includes all
the hdfs/yarn/mapreduce/hdds binaries and libraries. (Jar files are put in separated directory).
> The hdds components and hdfs compobebts all could be started from the bin. 
> *Proposed version*
> Create a sepearated hadoop-dist/ozone-2.1.0 which contains only the hdfs AND hdds components.
Both the hdfs namenode and hdds datanode/scm/ksm could be started from the ozone-2.1.0 package.

> Hdds packages would be removed from the original hadoop-3.2.0 directory.
> This is a relatively small change. On further JIRAs we need to :
>  * Create a shaded datanode plugin which could be used with any existing hadoop cluster
>  * Use standalone ObjectStore/Ozone server instead of the Namenode+Datanod plugin.
>  * Add test cases for both the ozone-only and the mixed clusters (ozone + hdfs)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message