hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-922) Create isolated classloder to use ozonefs with any older hadoop versions
Date Fri, 01 Feb 2019 15:31:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758410#comment-16758410
] 

Hadoop QA commented on HDDS-922:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 25s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  0s{color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 22s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 52s{color}
| {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  0s{color} | {color:blue}
Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m  0s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 11s{color} |
{color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 40s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 57s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m  0s{color}
| {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  0s{color} | {color:red}
The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>.
Refer https://git-scm.com/docs/git-apply {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  0s{color} | {color:blue}
Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m  0s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 18s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 44s{color} | {color:red}
hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 44s{color} | {color:green}
hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 33s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 36s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient |
|   | hadoop.ozone.client.rpc.TestFailureHandlingByClient |
|   | hadoop.ozone.om.TestScmChillMode |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-922 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12957267/HDDS-922.004.patch
|
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  checkstyle  shellcheck  |
| uname | Linux 6f13c5d0740b 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / 13aa939 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| shellcheck | v0.4.6 |
| whitespace | https://builds.apache.org/job/PreCommit-HDDS-Build/2166/artifact/out/whitespace-eol.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDDS-Build/2166/artifact/out/patch-unit-hadoop-ozone.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HDDS-Build/2166/testReport/ |
| Max. process+thread count | 1120 (vs. ulimit of 10000) |
| modules | C: hadoop-hdds/common hadoop-hdds/docs hadoop-ozone hadoop-ozone/dist hadoop-ozone/ozonefs
hadoop-ozone/ozonefs-lib hadoop-ozone/ozonefs-lib-legacy U: . |
| Console output | https://builds.apache.org/job/PreCommit-HDDS-Build/2166/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Create isolated classloder to use ozonefs with any older hadoop versions
> ------------------------------------------------------------------------
>
>                 Key: HDDS-922
>                 URL: https://issues.apache.org/jira/browse/HDDS-922
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>          Components: Ozone Filesystem
>            Reporter: Elek, Marton
>            Assignee: Elek, Marton
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HDDS-922.001.patch, HDDS-922.002.patch, HDDS-922.003.patch, HDDS-922.004.patch
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> As of now we create a shaded ozonefs artifact which includes all the required class files
to use ozonefs (Hadoop compatible file system for Ozone)
> But the shading process of this artifact is very easy, it includes all the class files
but no relocation rules (package name renaming) are configured. With this approach ozonefs
can be used from the compatible hadoop version (this is hadoop 3.1 only, I guess) but can't
be used with any older hadoop version as it requires the newer version of hadoop-common.
> I tried to configure a full shading (with relocation) but it's not a simple task. For
example a pure (non-relocated) Configuration is required by the ozonefs itself, but an other,
newer Configuration class is required by the ozone client code which is a dependency of OzoneFileSystem
So we need a relocated and a non-relocated class in the same time.
> I tried out a different approach: I moved out all of the ozone specific classes from
the OzoneFileSystem to an adapter class (OzoneClientAdapter). In case of an older hadoop version
the adapter class itself can be loaded with an isolated classloader. The isolated classloader
can load all the required classes from the jar file from a specific path. It doesn't require
any specific package relocation as the default class loader doesn't load these classes. 
> The OzoneFileSystem (in case of older hadoop version) can load the adapter with the isolated
classloader and only a few classes should be shared between the normal and isolated classloader
(the interface of the adapter and the types in the method signatures). All of the other ozone
classes and the newer hadoop dependencies will be hidden by the isolated classloader.
> This patch is more like a proof of concept, I would like to start a discussion about
this approach. I successfully used the generated artifact to use ozonefs from spark 2.4 default
distribution (which includes hadoop 2.7). 
> For a final patch I would add some check to use the ozonefs without any classpath separation
by default. (could be configured or chosen by automatically)
> For using spark (+ hadoop 2.7 + kubernetes scheduler) together with ozone, you can check
this screencast: https://www.youtube.com/watch?v=cpRJcSHIEdM&t=8s



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message